Merge branch 'dev/octavius-catto' of github.com:fishtown-analytics/db…

…t into fix/handle-query-array-types

CHANGELOG.md

@@ -1,4 +1,10 @@
 ## dbt next (release TBD)
+
+### Breaking changes
+- Added a new dbt_project.yml version format. This emits a deprecation warning currently, but support for the existing version will be removed in a future dbt version ([#2300](https:/fishtown-analytics/dbt/issues/2300), [#2312](https:/fishtown-analytics/dbt/pull/2312))
+- The `graph` object available in some dbt contexts now has an additional member `sources` (along side the existing `nodes`). Sources have been removed from `nodes` and added to `sources` instead ([#2312](https:/fishtown-analytics/dbt/pull/2312))
+- The 'location' field has been removed from bigquery catalogs ([#2382](https:/fishtown-analytics/dbt/pull/2382))
+
 ### Features
 - Added --fail-fast argument for dbt run and dbt test to fail on first test failure or runtime error. ([#1649](https:/fishtown-analytics/dbt/issues/1649), [#2224](https:/fishtown-analytics/dbt/pull/2224))
 - Support for appending query comments to SQL queries. ([#2138](https:/fishtown-analytics/dbt/issues/2138), [#2199](https:/fishtown-analytics/dbt/pull/2199))
@@ -9,7 +15,13 @@
 - Users can supply paths as arguments to `--models` and `--select`, either explicitily by prefixing with `path:` or implicitly with no prefix. ([#454](https:/fishtown-analytics/dbt/issues/454), [#2258](https:/fishtown-analytics/dbt/pull/2258))
 - dbt now builds the relation cache for "dbt compile" and "dbt ls" as well as "dbt run" ([#1705](https:/fishtown-analytics/dbt/issues/1705), [#2319](https:/fishtown-analytics/dbt/pull/2319))
 - Snowflake now uses "show terse objects" to build the relations cache instead of selecting from the information schema ([#2174](https:/fishtown-analytics/dbt/issues/2174), [#2322](https:/fishtown-analytics/dbt/pull/2322))
+- Snowflake now uses "describe table" to get the columns in a relation ([#2260](https:/fishtown-analytics/dbt/issues/2260), [#2324](https:/fishtown-analytics/dbt/pull/2324))
+- Add a 'depends_on' attribute to the log record extra field ([#2316](https:/fishtown-analytics/dbt/issues/2316), [#2341](https:/fishtown-analytics/dbt/pull/2341))
+- Added a '--no-browser' argument to "dbt docs serve" so you can serve docs in an environment that only has a CLI browser which would otherwise deadlock dbt ([#2004](https:/fishtown-analytics/dbt/issues/2004), [#2364](https:/fishtown-analytics/dbt/pull/2364))
 - Snowflake now uses "describe table" to get the columns in a relation ([#2260](https:/fishtown-analytics/dbt/issues/2260), [#2324](https:/fishtown-analytics/dbt/pull/2324))
+- Sources (and therefore freshness tests) can be enabled and disabled via dbt_project.yml ([#2283](https:/fishtown-analytics/dbt/issues/2283), [#2312](https:/fishtown-analytics/dbt/pull/2312), [#2357](https:/fishtown-analytics/dbt/pull/2357))
+- schema.yml files are now fully rendered in a context that is aware of vars declared in from dbt_project.yml files ([#2269](https:/fishtown-analytics/dbt/issues/2269), [#2357](https:/fishtown-analytics/dbt/pull/2357))
+- Sources from dependencies can be overridden in schema.yml files ([#2287](https:/fishtown-analytics/dbt/issues/2287), [#2357](https:/fishtown-analytics/dbt/pull/2357))
 
 ### Fixes
 - When a jinja value is undefined, give a helpful error instead of failing with cryptic "cannot pickle ParserMacroCapture" errors ([#2110](https:/fishtown-analytics/dbt/issues/2110), [#2184](https:/fishtown-analytics/dbt/pull/2184))
@@ -22,6 +34,11 @@
 - Return error message when profile is empty in profiles.yml. ([#2292](https:/fishtown-analytics/dbt/issues/2292), [#2297](https:/fishtown-analytics/dbt/pull/2297))
 - Fix skipped node count in stdout at the end of a run ([#2095](https:/fishtown-analytics/dbt/issues/2095), [#2310](https:/fishtown-analytics/dbt/pull/2310))
 - Fix an issue where BigQuery incorrectly used a relation's quote policy as the basis for the information schema's include policy, instead of the relation's include policy. ([#2188](https:/fishtown-analytics/dbt/issues/2188), [#2325](https:/fishtown-analytics/dbt/pull/2325))
+- Fix "dbt deps" command so it respects the "--project-dir" arg if specified. ([#2338](https:/fishtown-analytics/dbt/issues/2338), [#2339](https:/fishtown-analytics/dbt/issues/2339))
+- On `run_cli` API calls that are passed `--vars` differing from the server's `--vars`, the RPC server rebuilds the manifest for that call. ([#2265](https:/fishtown-analytics/dbt/issues/2265), [#2363](https:/fishtown-analytics/dbt/pull/2363))
+- Fix "Object of type Decimal is not JSON serializable" error when BigQuery queries returned numeric types in nested data structures ([#2336](https:/fishtown-analytics/dbt/issues/2336), [#2348](https:/fishtown-analytics/dbt/pull/2348))
+- No longer query the information_schema.schemata view on bigquery ([#2320](https:/fishtown-analytics/dbt/issues/2320), [#2382](https:/fishtown-analytics/dbt/pull/2382))
+- Add support for `sql_header` config in incremental models ([#2136](https:/fishtown-analytics/dbt/issues/2136), [#2200](https:/fishtown-analytics/dbt/pull/2200))
 
 ### Under the hood
 - Added more tests for source inheritance ([#2264](https:/fishtown-analytics/dbt/issues/2264), [#2291](https:/fishtown-analytics/dbt/pull/2291))
@@ -33,6 +50,8 @@ Contributors:
  - [@jeremyyeo](https:/jeremyyeo) [#2259](https:/fishtown-analytics/dbt/pull/2259)
  - [@rodrigodelmonte](https:/rodrigodelmonte) [#2298](https:/fishtown-analytics/dbt/pull/2298)
  - [@sumanau7](https:/sumanau7) ([#2279](https:/fishtown-analytics/dbt/pull/2279), [#2263](https:/fishtown-analytics/dbt/pull/2263), [#2297](https:/fishtown-analytics/dbt/pull/2297))
+ - [@nickwu241](https:/nickwu241) [#2339](https:/fishtown-analytics/dbt/issues/2339)
+ - [@Fokko](https:/Fokko) [#2361](https:/fishtown-analytics/dbt/pull/2361)
 
 ## dbt 0.16.1 (April 14, 2020)
 
@@ -572,6 +591,7 @@ Over a dozen contributors wrote code for this release of dbt! Thanks for taking
 - [@josegalarza](https:/josegalarza) ([#1571](https:/fishtown-analytics/dbt/pull/1571))
 - [@rmgpinto](https:/rmgpinto) ([docs#31](https:/fishtown-analytics/dbt-docs/pull/31), [docs#32](https:/fishtown-analytics/dbt-docs/pull/32))
 - [@groodt](https:/groodt) ([docs#34](https:/fishtown-analytics/dbt-docs/pull/34))
+- [@dcereijodo](https:/dcereijodo) ([#2341](https:/fishtown-analytics/dbt/pull/2341))
 
 
 ## dbt 0.13.1 (May 13, 2019)

core/dbt/adapters/base/__init__.py

@@ -10,5 +10,5 @@
  SchemaSearchMap,
 )
 from dbt.adapters.base.column import Column # noqa
-from dbt.adapters.base.impl import BaseAdapter # noqa
+from dbt.adapters.base.impl import AdapterConfig, BaseAdapter # noqa
 from dbt.adapters.base.plugin import AdapterPlugin # noqa

core/dbt/adapters/base/impl.py

@@ -1,10 +1,12 @@
 import abc
 from concurrent.futures import as_completed, Future
 from contextlib import contextmanager
+from dataclasses import dataclass
 from datetime import datetime
+from itertools import chain
 from typing import (
- Optional, Tuple, Callable, Iterable, FrozenSet, Type, Dict, Any, List,
- Mapping, Iterator, Union, Set
+ Optional, Tuple, Callable, Iterable, Type, Dict, Any, List, Mapping,
+ Iterator, Union, Set
 )
 
 import agate
@@ -23,6 +25,7 @@
 from dbt.contracts.graph.compiled import CompileResultNode, CompiledSeedNode
 from dbt.contracts.graph.manifest import Manifest
 from dbt.contracts.graph.parsed import ParsedSeedNode
+from dbt.contracts.graph.model_config import BaseConfig
 from dbt.exceptions import warn_or_error
 from dbt.node_types import NodeType
 from dbt.logger import GLOBAL_LOGGER as logger
@@ -105,6 +108,11 @@ def _relation_name(rel: Optional[BaseRelation]) -> str:
  return str(rel)
 
 
+@dataclass
+class AdapterConfig(BaseConfig):
+ pass
+
+
 class BaseAdapter(metaclass=AdapterMeta):
  """The BaseAdapter provides an abstract base class for adapters.
 
@@ -147,7 +155,7 @@ class BaseAdapter(metaclass=AdapterMeta):
 
  # A set of clobber config fields accepted by this adapter
  # for use in materializations
- AdapterSpecificConfigs: FrozenSet[str] = frozenset()
+ AdapterSpecificConfigs: Type[AdapterConfig] = AdapterConfig
 
  def __init__(self, config):
  self.config = config
@@ -282,7 +290,11 @@ def _get_cache_schemas(
  lowercase strings.
  """
  info_schema_name_map = SchemaSearchMap()
- for node in manifest.nodes.values():
+ nodes: Iterator[CompileResultNode] = chain(
+ manifest.nodes.values(),
+ manifest.sources.values(),
+ )
+ for node in nodes:
  if exec_only and node.resource_type not in NodeType.executable():
  continue
  relation = self.Relation.create_from(self.config, node)

core/dbt/adapters/factory.py

@@ -7,7 +7,7 @@
 from dbt.logger import GLOBAL_LOGGER as logger
 from dbt.contracts.connection import Credentials, AdapterRequiredConfig
 
-from dbt.adapters.base.impl import BaseAdapter
+from dbt.adapters.base.impl import BaseAdapter, AdapterConfig
 from dbt.adapters.base.plugin import AdapterPlugin
 
 
@@ -40,6 +40,12 @@ def get_relation_class_by_name(self, name: str) -> Type[BaseRelation]:
  adapter = self.get_adapter_class_by_name(name)
  return adapter.Relation
 
+ def get_config_class_by_name(
+ self, name: str
+ ) -> Type[AdapterConfig]:
+ adapter = self.get_adapter_class_by_name(name)
+ return adapter.AdapterSpecificConfigs
+
  def load_plugin(self, name: str) -> Type[Credentials]:
  # this doesn't need a lock: in the worst case we'll overwrite PACKAGES
  # and adapter_type entries with the same value, as they're all
@@ -137,6 +143,10 @@ def get_adapter_class_by_name(name: str) -> Type[BaseAdapter]:
  return FACTORY.get_adapter_class_by_name(name)
 
 
+def get_config_class_by_name(name: str) -> Type[AdapterConfig]:
+ return FACTORY.get_config_class_by_name(name)
+
+
 def get_relation_class_by_name(name: str) -> Type[BaseRelation]:
  return FACTORY.get_relation_class_by_name(name)
 

core/dbt/clients/agate_helper.py

@@ -4,6 +4,7 @@
 import datetime
 import isodate
 import json
+import dbt.utils
 from typing import Iterable, List, Dict, Union, Optional, Any
 
 from dbt.exceptions import RuntimeException
@@ -92,7 +93,7 @@ def table_from_data_flat(data, column_names: Iterable[str]) -> agate.Table:
  row = []
  for value in list(_row.values()):
  if isinstance(value, (dict, list, tuple)):
- row.append(json.dumps(value))
+ row.append(json.dumps(value, cls=dbt.utils.JSONEncoder))
  else:
  row.append(value)
  rows.append(row)

core/dbt/clients/jinja.py

@@ -4,7 +4,9 @@
 import re
 import tempfile
 import threading
+from ast import literal_eval
 from contextlib import contextmanager
+from itertools import chain, islice
 from typing import (
  List, Union, Set, Optional, Dict, Any, Iterator, Type, NoReturn
 )
@@ -102,9 +104,51 @@ class NativeSandboxEnvironment(MacroFuzzEnvironment):
  code_generator_class = jinja2.nativetypes.NativeCodeGenerator
 
 
+def quoted_native_concat(nodes):
+ """This is almost native_concat from the NativeTemplate, except in the
+ special case of a single argument that is a quoted string and returns a
+ string, the quotes are re-inserted.
+ """
+ head = list(islice(nodes, 2))
+
+ if not head:
+ return None
+
+ if len(head) == 1:
+ raw = head[0]
+ else:
+ raw = "".join([str(v) for v in chain(head, nodes)])
+
+ try:
+ result = literal_eval(raw)
+ except (ValueError, SyntaxError, MemoryError):
+ return raw
+
+ if len(head) == 1 and len(raw) > 2 and isinstance(result, str):
+ return _requote_result(raw, result)
+ else:
+ return result
+
+
 class NativeSandboxTemplate(jinja2.nativetypes.NativeTemplate): # mypy: ignore
  environment_class = NativeSandboxEnvironment
 
+ def render(self, *args, **kwargs):
+ """Render the template to produce a native Python type. If the
+ result is a single node, its value is returned. Otherwise, the
+ nodes are concatenated as strings. If the result can be parsed
+ with :func:`ast.literal_eval`, the parsed value is returned.
+ Otherwise, the string is returned.
+ """
+ vars = dict(*args, **kwargs)
+
+ try:
+ return quoted_native_concat(
+ self.root_render_func(self.new_context(vars))
+ )
+ except Exception:
+ return self.environment.handle_exception()
+
 
 NativeSandboxEnvironment.template_class = NativeSandboxTemplate # type: ignore
 
@@ -425,7 +469,7 @@ def render_template(template, ctx: Dict[str, Any], node=None) -> str:
  return template.render(ctx)
 
 
-def _requote_result(raw_value, rendered):
+def _requote_result(raw_value: str, rendered: str) -> str:
  double_quoted = raw_value.startswith('"') and raw_value.endswith('"')
  single_quoted = raw_value.startswith("'") and raw_value.endswith("'")
  if double_quoted:
@@ -451,12 +495,7 @@ def get_rendered(
  capture_macros=capture_macros,
  native=native,
  )
-
- result = render_template(template, ctx, node)
-
- if native and isinstance(result, str):
- result = _requote_result(string, result)
- return result
+ return render_template(template, ctx, node)
 
 
 def undefined_error(msg) -> NoReturn:

core/dbt/compilation.py

@@ -1,4 +1,3 @@
-import itertools
 import os
 from collections import defaultdict
 from typing import List, Dict, Any
@@ -11,8 +10,8 @@
 from dbt.linker import Linker
 
 from dbt.context.providers import generate_runtime_model
+from dbt.contracts.graph.compiled import NonSourceNode
 from dbt.contracts.graph.manifest import Manifest
-import dbt.contracts.project
 import dbt.exceptions
 import dbt.flags
 import dbt.config
@@ -61,22 +60,24 @@ def print_compile_stats(stats):
  logger.info("Found {}".format(stat_line))
 
 
-def _node_enabled(node):
+def _node_enabled(node: NonSourceNode):
  # Disabled models are already excluded from the manifest
  if node.resource_type == NodeType.Test and not node.config.enabled:
  return False
  else:
  return True
 
 
-def _generate_stats(manifest):
+def _generate_stats(manifest: Manifest):
  stats: Dict[NodeType, int] = defaultdict(int)
- for node_name, node in itertools.chain(
- manifest.nodes.items(),
- manifest.macros.items()):
+ for node in manifest.nodes.values():
  if _node_enabled(node):
  stats[node.resource_type] += 1
 
+ for source in manifest.sources.values():
+ stats[source.resource_type] += 1
+ for macro in manifest.macros.values():
+ stats[macro.resource_type] += 1
  return stats
 
 
@@ -183,24 +184,34 @@ def compile_node(self, node, manifest, extra_context=None):
 
  return injected_node
 
- def write_graph_file(self, linker, manifest):
+ def write_graph_file(self, linker: Linker, manifest: Manifest):
  filename = graph_file_name
  graph_path = os.path.join(self.config.target_path, filename)
  if dbt.flags.WRITE_JSON:
  linker.write_graph(graph_path, manifest)
 
- def link_node(self, linker, node, manifest):
+ def link_node(
+ self, linker: Linker, node: NonSourceNode, manifest: Manifest
+ ):
  linker.add_node(node.unique_id)
 
  for dependency in node.depends_on_nodes:
- if manifest.nodes.get(dependency):
+ if dependency in manifest.nodes:
  linker.dependency(
  node.unique_id,
- (manifest.nodes.get(dependency).unique_id))
+ (manifest.nodes[dependency].unique_id)
+ )
+ elif dependency in manifest.sources:
+ linker.dependency(
+ node.unique_id,
+ (manifest.sources[dependency].unique_id)
+ )
  else:
  dbt.exceptions.dependency_not_found(node, dependency)
 
- def link_graph(self, linker, manifest):
+ def link_graph(self, linker: Linker, manifest: Manifest):
+ for source in manifest.sources.values():
+ linker.add_node(source.unique_id)
  for node in manifest.nodes.values():
  self.link_node(linker, node, manifest)
 
@@ -209,7 +220,7 @@ def link_graph(self, linker, manifest):
  if cycle:
  raise RuntimeError("Found a cycle: {}".format(cycle))
 
- def compile(self, manifest, write=True):
+ def compile(self, manifest: Manifest, write=True):
  linker = Linker()
 
  self.link_graph(linker, manifest)

core/dbt/config/__init__.py

@@ -2,4 +2,3 @@
 from .profile import Profile, PROFILES_DIR, read_user_config # noqa
 from .project import Project # noqa
 from .runtime import RuntimeConfig, UnsetProfileConfig # noqa
-from .renderer import ConfigRenderer # noqa