-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CT-634] Node configs: tech debt #5236
Labels
design_needed
tech_debt
Behind-the-scenes changes, with little direct impact on end-user functionality
Comments
jtcohen6
added
tech_debt
Behind-the-scenes changes, with little direct impact on end-user functionality
Team:Language
labels
May 12, 2022
github-actions
bot
changed the title
Node configs: tech debt
[CT-634] Node configs: tech debt
May 12, 2022
This was referenced Aug 9, 2022
[CT-676] Set default table type to
parquet
(instead of implicit default text
)
dbt-labs/dbt-spark#363
Closed
This was referenced Dec 7, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
design_needed
tech_debt
Behind-the-scenes changes, with little direct impact on end-user functionality
Context
grants
as a config underNodeConfig
, where it gets inherited by node types that won't actually use it.Problem
We need to better organize our tightly bundled
NodeConfig
. Many of these configs are only relevant to "executable" nodes (models, snapshots, seeds, and tests IFFstore_failures
is enabled). Some of them are only relevant to incremental models in particular. Analyses have no business withdatabase
/schema
/alias
.Existing proposals
Incremental models. Many many configs are really just relevant to the
incremental
model materialization. As discussed in #5089: It feels a little icky that configs likeunique_key
andon_schema_change
appear in the config for every node, when they're really just relevant to incremental models in particular. Could we isolate these to appear on incremental models only? This could be:IncrementalModelConfig
subclass, which would supplyunique_key
,on_schema_change
,incremental_strategy
, and (new)incremental_predicates
if and only ifmaterialized == 'incremental'
incremental
that containsstrategy
+predicates
, and the old names can be passthroughs to that dictionary for backwards compatibilityMetadata interface. Every time we add a new attribute to
NodeConfig
, it changes themanifest.json
artifact schema. As discussed in #4617: Given that users can supply whatever configs they want, to any node type that acceptsconfig
, should we even consider this a contracted metadata interface? I think the external contract fornode.config
should really beDict[str, Any]
, and we should always be in the habit of "promoting" configs that are relevant and reliable to downstream metadata consumers as top-level node keys. Similar to what we did withnode_info
on logging events, which "promotes"config.materialized
intonode_info.materialized
.Adapter-specific configs (sort of discussed in #4622): I'm actually not sure if
AdapterConfig
works at all today. My sense is that their keys don't show up in the manifest unless set, their types aren't really validated until used (at runtime), and their default values don't get passed through. The code here is kinda jank, to put it nicely.The text was updated successfully, but these errors were encountered: