Speed up Snowflake column comments, while still avoiding errors #3543
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
resolves #3541, slack thread
see also: #3149, #3039
Description
#3149 did a wonderful job of resolving the undesirable behavior (originally reported in #3039) whereby a column description, specified in yaml, would cause a model to fail building if the column doesn't actually exist in the model query. This "strictness" could be a feature, someday, but it's not how we think about properties (and especially descriptions) today.
Unfortunately, the approach taken in #3149 is resulting in much, much slower builds for models with many description-bearing columns—from seconds to minutes. This is because the Snowflake python connector requires running each semicolon-delineated query on its own, and so they run in sequence, hundreds of times. (Whereas
dbt-postgres
, which also uses per-columncomment on
statements, can run big batches of semicolon-delineated statements all at once.)This PR takes an alternative approach—option 2 that I outlined in the original issue—by checking to see which columns exist in the just-created table, via
adapter.get_columns_in_relation
, and then comparing the result against the dictionary of columns with descriptions defined.I'd like to sneak this into v0.20.0 final if possible :)
Checklist
CHANGELOG.md
and added information about my change to the "dbt next" section.