Merge branch 'dev/octavius-catto' into column_comments

CHANGELOG.md

@@ -1,4 +1,10 @@
 ## dbt next (release TBD)
+
+### Breaking changes
+- Added a new dbt_project.yml version format. This emits a deprecation warning currently, but support for the existing version will be removed in a future dbt version ([#2300](https:/fishtown-analytics/dbt/issues/2300), [#2312](https:/fishtown-analytics/dbt/pull/2312))
+- The `graph` object available in some dbt contexts now has an additional member `sources` (along side the existing `nodes`). Sources have been removed from `nodes` and added to `sources` instead ([#2312](https:/fishtown-analytics/dbt/pull/2312))
+- The 'location' field has been removed from bigquery catalogs ([#2382](https:/fishtown-analytics/dbt/pull/2382))
+
 ### Features
 - Added --fail-fast argument for dbt run and dbt test to fail on first test failure or runtime error. ([#1649](https:/fishtown-analytics/dbt/issues/1649), [#2224](https:/fishtown-analytics/dbt/pull/2224))
 - Support for appending query comments to SQL queries. ([#2138](https:/fishtown-analytics/dbt/issues/2138), [#2199](https:/fishtown-analytics/dbt/pull/2199))
@@ -10,8 +16,14 @@
 - dbt now builds the relation cache for "dbt compile" and "dbt ls" as well as "dbt run" ([#1705](https:/fishtown-analytics/dbt/issues/1705), [#2319](https:/fishtown-analytics/dbt/pull/2319))
 - Snowflake now uses "show terse objects" to build the relations cache instead of selecting from the information schema ([#2174](https:/fishtown-analytics/dbt/issues/2174), [#2322](https:/fishtown-analytics/dbt/pull/2322))
 - Snowflake now uses "describe table" to get the columns in a relation ([#2260](https:/fishtown-analytics/dbt/issues/2260), [#2324](https:/fishtown-analytics/dbt/pull/2324))
+- Add a 'depends_on' attribute to the log record extra field ([#2316](https:/fishtown-analytics/dbt/issues/2316), [#2341](https:/fishtown-analytics/dbt/pull/2341))
 - Added a '--no-browser' argument to "dbt docs serve" so you can serve docs in an environment that only has a CLI browser which would otherwise deadlock dbt ([#2004](https:/fishtown-analytics/dbt/issues/2004), [#2364](https:/fishtown-analytics/dbt/pull/2364))
 - Relation comments supported for Snowflake tables and views. Column comments supported for tables. ([#1722](https:/fishtown-analytics/dbt/issues/1722), [#2321](https:/fishtown-analytics/dbt/pull/2321))
+- Snowflake now uses "describe table" to get the columns in a relation ([#2260](https:/fishtown-analytics/dbt/issues/2260), [#2324](https:/fishtown-analytics/dbt/pull/2324))
+- Sources (and therefore freshness tests) can be enabled and disabled via dbt_project.yml ([#2283](https:/fishtown-analytics/dbt/issues/2283), [#2312](https:/fishtown-analytics/dbt/pull/2312), [#2357](https:/fishtown-analytics/dbt/pull/2357))
+- schema.yml files are now fully rendered in a context that is aware of vars declared in from dbt_project.yml files ([#2269](https:/fishtown-analytics/dbt/issues/2269), [#2357](https:/fishtown-analytics/dbt/pull/2357))
+- Sources from dependencies can be overridden in schema.yml files ([#2287](https:/fishtown-analytics/dbt/issues/2287), [#2357](https:/fishtown-analytics/dbt/pull/2357))
+- Implement persist_docs for both `relation` and `comments` on postgres and redshift, and extract them when getting the catalog. ([#2333](https:/fishtown-analytics/dbt/issues/2333), [#2378](https:/fishtown-analytics/dbt/pull/2378))
 
 ### Fixes
 - When a jinja value is undefined, give a helpful error instead of failing with cryptic "cannot pickle ParserMacroCapture" errors ([#2110](https:/fishtown-analytics/dbt/issues/2110), [#2184](https:/fishtown-analytics/dbt/pull/2184))
@@ -26,7 +38,16 @@
 - Fix an issue where BigQuery incorrectly used a relation's quote policy as the basis for the information schema's include policy, instead of the relation's include policy. ([#2188](https:/fishtown-analytics/dbt/issues/2188), [#2325](https:/fishtown-analytics/dbt/pull/2325))
 - Fix "dbt deps" command so it respects the "--project-dir" arg if specified. ([#2338](https:/fishtown-analytics/dbt/issues/2338), [#2339](https:/fishtown-analytics/dbt/issues/2339))
 - On `run_cli` API calls that are passed `--vars` differing from the server's `--vars`, the RPC server rebuilds the manifest for that call. ([#2265](https:/fishtown-analytics/dbt/issues/2265), [#2363](https:/fishtown-analytics/dbt/pull/2363))
+- Remove the query job SQL from bigquery exceptions ([#2383](https:/fishtown-analytics/dbt/issues/2383), [#2393](https:/fishtown-analytics/dbt/pull/2393))
 - Fix "Object of type Decimal is not JSON serializable" error when BigQuery queries returned numeric types in nested data structures ([#2336](https:/fishtown-analytics/dbt/issues/2336), [#2348](https:/fishtown-analytics/dbt/pull/2348))
+- No longer query the information_schema.schemata view on bigquery ([#2320](https:/fishtown-analytics/dbt/issues/2320), [#2382](https:/fishtown-analytics/dbt/pull/2382))
+- Preserve original subdirectory structure in compiled files. ([#2173](https:/fishtown-analytics/dbt/issues/2173), [#2349](https:/fishtown-analytics/dbt/pull/2349))
+- Add support for `sql_header` config in incremental models ([#2136](https:/fishtown-analytics/dbt/issues/2136), [#2200](https:/fishtown-analytics/dbt/pull/2200))
+- The ambiguous alias check now examines the node's database value as well as the schema/identifier ([#2326](https:/fishtown-analytics/dbt/issues/2326), [#2387](https:/fishtown-analytics/dbt/pull/2387))
+- Postgres array types can now be returned via `run_query` macro calls ([#2337](https:/fishtown-analytics/dbt/issues/2337), [#2376](https:/fishtown-analytics/dbt/pull/2376))
+- Add missing comma to `dbt compile` help text ([#2388](https:/fishtown-analytics/dbt/issues/2388) [#2389](https:/fishtown-analytics/dbt/pull/2389))
+- Fix for non-atomic snapshot staging table creation ([#1884](https:/fishtown-analytics/dbt/issues/1884), [#2390](https:/fishtown-analytics/dbt/pull/2390))
+- Fix for snapshot errors when strategy changes from `check` to `timestamp` between runs ([#2350](https:/fishtown-analytics/dbt/issues/2350), [#2391](https:/fishtown-analytics/dbt/pull/2391))
 
 ### Under the hood
 - Added more tests for source inheritance ([#2264](https:/fishtown-analytics/dbt/issues/2264), [#2291](https:/fishtown-analytics/dbt/pull/2291))
@@ -41,6 +62,8 @@ Contributors:
  - [@nickwu241](https:/nickwu241) [#2339](https:/fishtown-analytics/dbt/issues/2339)
  - [@Fokko](https:/Fokko) [#2361](https:/fishtown-analytics/dbt/pull/2361)
  - [@snowflakeseitz](https:/snowflakeseitz) [#2321](https:/fishtown-analytics/dbt/pull/2321)
+ - [@franloza](https:/franloza) [#2349](https:/fishtown-analytics/dbt/pull/2349)
+ - [@sethwoodworth](https:/sethwoodworth) [#2389](https:/fishtown-analytics/dbt/pull/2389)
 
 ## dbt 0.16.1 (April 14, 2020)
 
@@ -580,6 +603,7 @@ Over a dozen contributors wrote code for this release of dbt! Thanks for taking
 - [@josegalarza](https:/josegalarza) ([#1571](https:/fishtown-analytics/dbt/pull/1571))
 - [@rmgpinto](https:/rmgpinto) ([docs#31](https:/fishtown-analytics/dbt-docs/pull/31), [docs#32](https:/fishtown-analytics/dbt-docs/pull/32))
 - [@groodt](https:/groodt) ([docs#34](https:/fishtown-analytics/dbt-docs/pull/34))
+- [@dcereijodo](https:/dcereijodo) ([#2341](https:/fishtown-analytics/dbt/pull/2341))
 
 
 ## dbt 0.13.1 (May 13, 2019)

core/dbt/adapters/base/impl.py

@@ -3,6 +3,7 @@
 from contextlib import contextmanager
 from dataclasses import dataclass
 from datetime import datetime
+from itertools import chain
 from typing import (
  Optional, Tuple, Callable, Iterable, Type, Dict, Any, List, Mapping,
  Iterator, Union, Set
@@ -289,7 +290,11 @@ def _get_cache_schemas(
  lowercase strings.
  """
  info_schema_name_map = SchemaSearchMap()
- for node in manifest.nodes.values():
+ nodes: Iterator[CompileResultNode] = chain(
+ manifest.nodes.values(),
+ manifest.sources.values(),
+ )
+ for node in nodes:
  if exec_only and node.resource_type not in NodeType.executable():
  continue
  relation = self.Relation.create_from(self.config, node)

core/dbt/adapters/sql/connections.py

@@ -107,7 +107,10 @@ def get_result_from_cursor(cls, cursor: Any) -> agate.Table:
  rows = cursor.fetchall()
  data = cls.process_results(column_names, rows)
 
- return dbt.clients.agate_helper.table_from_data(data, column_names)
+ return dbt.clients.agate_helper.table_from_data_flat(
+ data,
+ column_names
+ )
 
  def execute(
  self, sql: str, auto_begin: bool = False, fetch: bool = False

core/dbt/clients/_jinja_blocks.py

@@ -92,6 +92,18 @@ def __init__(self, data):
  self._parenthesis_stack = []
  self.pos = 0
 
+ def linepos(self, end=None) -> str:
+ """Given an absolute position in the input data, return a pair of
+ line number + relative position to the start of the line.
+ """
+ end_val: int = self.pos if end is None else end
+ data = self.data[:end_val]
+ # if not found, rfind returns -1, and -1+1=0, which is perfect!
+ last_line_start = data.rfind('\n') + 1
+ # it's easy to forget this, but line numbers are 1-indexed
+ line_number = data.count('\n') + 1
+ return f'{line_number}:{end_val - last_line_start}'
+
  def advance(self, new_position):
  self.pos = new_position
 
@@ -320,20 +332,28 @@ def find_blocks(self, allowed_blocks=None, collect_raw_data=True):
  dbt.exceptions.raise_compiler_error((
  'Got an unexpected control flow end tag, got {} but '
  'never saw a preceeding {} (@ {})'
- ).format(tag.block_type_name, expected, tag.start))
+ ).format(
+ tag.block_type_name,
+ expected,
+ self.tag_parser.linepos(tag.start)
+ ))
  expected = _CONTROL_FLOW_TAGS[found]
  if expected != tag.block_type_name:
  dbt.exceptions.raise_compiler_error((
  'Got an unexpected control flow end tag, got {} but '
  'expected {} next (@ {})'
- ).format(tag.block_type_name, expected, tag.start))
+ ).format(
+ tag.block_type_name,
+ expected,
+ self.tag_parser.linepos(tag.start)
+ ))
 
  if tag.block_type_name in allowed_blocks:
  if self.stack:
  dbt.exceptions.raise_compiler_error((
  'Got a block definition inside control flow at {}. '
  'All dbt block definitions must be at the top level'
- ).format(tag.start))
+ ).format(self.tag_parser.linepos(tag.start)))
  if self.current is not None:
  dbt.exceptions.raise_compiler_error(
  duplicate_tags.format(outer=self.current, inner=tag)

core/dbt/clients/jinja.py

@@ -4,7 +4,9 @@
 import re
 import tempfile
 import threading
+from ast import literal_eval
 from contextlib import contextmanager
+from itertools import chain, islice
 from typing import (
  List, Union, Set, Optional, Dict, Any, Iterator, Type, NoReturn
 )
@@ -102,9 +104,51 @@ class NativeSandboxEnvironment(MacroFuzzEnvironment):
  code_generator_class = jinja2.nativetypes.NativeCodeGenerator
 
 
+def quoted_native_concat(nodes):
+ """This is almost native_concat from the NativeTemplate, except in the
+ special case of a single argument that is a quoted string and returns a
+ string, the quotes are re-inserted.
+ """
+ head = list(islice(nodes, 2))
+
+ if not head:
+ return None
+
+ if len(head) == 1:
+ raw = head[0]
+ else:
+ raw = "".join([str(v) for v in chain(head, nodes)])
+
+ try:
+ result = literal_eval(raw)
+ except (ValueError, SyntaxError, MemoryError):
+ return raw
+
+ if len(head) == 1 and len(raw) > 2 and isinstance(result, str):
+ return _requote_result(raw, result)
+ else:
+ return result
+
+
 class NativeSandboxTemplate(jinja2.nativetypes.NativeTemplate): # mypy: ignore
  environment_class = NativeSandboxEnvironment
 
+ def render(self, *args, **kwargs):
+ """Render the template to produce a native Python type. If the
+ result is a single node, its value is returned. Otherwise, the
+ nodes are concatenated as strings. If the result can be parsed
+ with :func:`ast.literal_eval`, the parsed value is returned.
+ Otherwise, the string is returned.
+ """
+ vars = dict(*args, **kwargs)
+
+ try:
+ return quoted_native_concat(
+ self.root_render_func(self.new_context(vars))
+ )
+ except Exception:
+ return self.environment.handle_exception()
+
 
 NativeSandboxEnvironment.template_class = NativeSandboxTemplate # type: ignore
 
@@ -425,7 +469,7 @@ def render_template(template, ctx: Dict[str, Any], node=None) -> str:
  return template.render(ctx)
 
 
-def _requote_result(raw_value, rendered):
+def _requote_result(raw_value: str, rendered: str) -> str:
  double_quoted = raw_value.startswith('"') and raw_value.endswith('"')
  single_quoted = raw_value.startswith("'") and raw_value.endswith("'")
  if double_quoted:
@@ -451,12 +495,7 @@ def get_rendered(
  capture_macros=capture_macros,
  native=native,
  )
-
- result = render_template(template, ctx, node)
-
- if native and isinstance(result, str):
- result = _requote_result(string, result)
- return result
+ return render_template(template, ctx, node)
 
 
 def undefined_error(msg) -> NoReturn:

core/dbt/compilation.py

@@ -1,4 +1,3 @@
-import itertools
 import os
 from collections import defaultdict
 from typing import List, Dict, Any
@@ -11,6 +10,7 @@
 from dbt.linker import Linker
 
 from dbt.context.providers import generate_runtime_model
+from dbt.contracts.graph.compiled import NonSourceNode
 from dbt.contracts.graph.manifest import Manifest
 import dbt.exceptions
 import dbt.flags
@@ -60,22 +60,24 @@ def print_compile_stats(stats):
  logger.info("Found {}".format(stat_line))
 
 
-def _node_enabled(node):
+def _node_enabled(node: NonSourceNode):
  # Disabled models are already excluded from the manifest
  if node.resource_type == NodeType.Test and not node.config.enabled:
  return False
  else:
  return True
 
 
-def _generate_stats(manifest):
+def _generate_stats(manifest: Manifest):
  stats: Dict[NodeType, int] = defaultdict(int)
- for node_name, node in itertools.chain(
- manifest.nodes.items(),
- manifest.macros.items()):
+ for node in manifest.nodes.values():
  if _node_enabled(node):
  stats[node.resource_type] += 1
 
+ for source in manifest.sources.values():
+ stats[source.resource_type] += 1
+ for macro in manifest.macros.values():
+ stats[macro.resource_type] += 1
  return stats
 
 
@@ -182,24 +184,34 @@ def compile_node(self, node, manifest, extra_context=None):
 
  return injected_node
 
- def write_graph_file(self, linker, manifest):
+ def write_graph_file(self, linker: Linker, manifest: Manifest):
  filename = graph_file_name
  graph_path = os.path.join(self.config.target_path, filename)
  if dbt.flags.WRITE_JSON:
  linker.write_graph(graph_path, manifest)
 
- def link_node(self, linker, node, manifest):
+ def link_node(
+ self, linker: Linker, node: NonSourceNode, manifest: Manifest
+ ):
  linker.add_node(node.unique_id)
 
  for dependency in node.depends_on_nodes:
- if manifest.nodes.get(dependency):
+ if dependency in manifest.nodes:
  linker.dependency(
  node.unique_id,
- (manifest.nodes.get(dependency).unique_id))
+ (manifest.nodes[dependency].unique_id)
+ )
+ elif dependency in manifest.sources:
+ linker.dependency(
+ node.unique_id,
+ (manifest.sources[dependency].unique_id)
+ )
  else:
  dbt.exceptions.dependency_not_found(node, dependency)
 
- def link_graph(self, linker, manifest):
+ def link_graph(self, linker: Linker, manifest: Manifest):
+ for source in manifest.sources.values():
+ linker.add_node(source.unique_id)
  for node in manifest.nodes.values():
  self.link_node(linker, node, manifest)
 
@@ -208,7 +220,7 @@ def link_graph(self, linker, manifest):
  if cycle:
  raise RuntimeError("Found a cycle: {}".format(cycle))
 
- def compile(self, manifest, write=True):
+ def compile(self, manifest: Manifest, write=True):
  linker = Linker()
 
  self.link_graph(linker, manifest)

core/dbt/config/project.py

@@ -575,7 +575,12 @@ def render_from_dict(
  rendered_project['project-root'] = project_root
  package_renderer = renderer.get_package_renderer()
  rendered_packages = package_renderer.render_data(packages_dict)
- return cls.from_project_config(rendered_project, rendered_packages)
+ try:
+ return cls.from_project_config(rendered_project, rendered_packages)
+ except DbtProjectError as exc:
+ if exc.path is None:
+ exc.path = os.path.join(project_root, 'dbt_project.yml')
+ raise
 
  @classmethod
  def partial_load(
@@ -645,6 +650,9 @@ def as_v1(self):
  # stuff any 'vars' entries into the old-style
  # models/seeds/snapshots dicts
  for project_name, items in dct['vars'].items():
+ if not isinstance(items, dict):
+ # can't translate top-level vars
+ continue
  for cfgkey in ['models', 'seeds', 'snapshots']:
  if project_name not in mutated[cfgkey]:
  mutated[cfgkey][project_name] = {}

core/dbt/config/renderer.py

@@ -35,7 +35,7 @@ def render_value(
  # if it wasn't read as a string, ignore it
  if not isinstance(value, str):
  return value
- return str(get_rendered(value, self.context))
+ return get_rendered(value, self.context, native=True)
 
  def render_data(
  self, data: Dict[str, Any]
@@ -131,17 +131,6 @@ class ProfileRenderer(BaseRenderer):
  def name(self):
  'Profile'
 
- def render_entry(self, value, keypath):
- result = super().render_entry(value, keypath)
-
- if len(keypath) == 1 and keypath[-1] == 'port':
- try:
- return int(result)
- except ValueError:
- # let the validator or connection handle this
- pass
- return result
-
 
 class SchemaYamlRenderer(BaseRenderer):
  DOCUMENTABLE_NODES = frozenset(