Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Override and disable sources in packages #2283

Closed
drewbanin opened this issue Apr 1, 2020 · 1 comment · Fixed by #2357
Closed

Override and disable sources in packages #2283

drewbanin opened this issue Apr 1, 2020 · 1 comment · Fixed by #2357
Labels
enhancement New feature or request packages Functionality for interacting with installed packages

Comments

@drewbanin
Copy link
Contributor

Describe the feature

If a source is located in a package, there is currently no way to disable that source to remove it from the manifest or the compilation context. To this end, sources should support an enabled config, as well as a first-class entry in the dbt_project.yml file.

Example syntax:

# models/airflow/sources.yml

sources:
  - name: snowplow
    enabled: false
    tables: [ ... ]

Source configurations (database, schema, enabled, freshness, etc) should be configurable via a sources: block in the dbt_project.yml file:

sources:
  my_source: # configure sources in the my_source package
    airflow: # configure sources in the airflow/ directory
      snowplow: # configure the snowplow source
        database: ....
        schema: ....
        enabled: true
        tables:
          event: # configure the snowplow.event source table
            loaded_at_field: collector_tstamp

Implementation notes:

  • The configs applied in a schema.yml file have historically been totally separate from configs applied in the dbt_project.yml. It would be a good thing to start bringing those two together, but for the scope of this issue, no sweeping changes are required for model configs.
  • The example above includes a hierarchical config based on file path. This would be consistent with how we handle configs for models (a good thing), but it is a tiny bit wonky (paths for .yml files are not incredibly well defined at the moment)
  • Source config precedence is similar to model config precedence
    1. in-package dbt_project.yml file
    2. in-package schema.yml file
    3. root-project dbt_project.yml file
    4. root-project schema.yml file
  • If a source is disabled, then source() calls referencing that source should fail and the source should not be represented in the manifest (akin to disabled models)
@drewbanin drewbanin added enhancement New feature or request packages Functionality for interacting with installed packages labels Apr 1, 2020
@drewbanin drewbanin added this to the Octavius Catto milestone Apr 1, 2020
@drewbanin
Copy link
Contributor Author

#2300 implements this for enabled configs on sources, @beckjake to additionally add support for source quoting configs in these blocks too

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request packages Functionality for interacting with installed packages
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant