-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue generating docs on BigQuery #1984
Comments
beckjake
added a commit
that referenced
this issue
Dec 13, 2019
…-info-schema-queries Fix bigquery catalog queries with nonexistent schemas (#1984)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Docs generation can fail in new and interesting ways in dbt v0.15.0. Whereas on most databases, the
information_schema
is accessed at the database-level, on BigQuery, it is sometimes accessed at theproject
level and other times at thedataset
level. This means that the catalog query on BigQuery can fail if a dataset does not exist. That is not the case on other plugins.I've seen some reports of users whose
dbt docs generate
task is failing on BigQuery withdataset xxx does not exist in region US
. This appears to be happening because dbt is querying the information schema for a dataset which does not exist.There are two ways this can happen:
The error that dbt encounters is:
This seems to indicate that dbt is checking a dataset called
dbt_ryan
which no models are configured to build into! Here, it's possible that dbt is errantly generating aninformation_schema
relation for a resource that should not participate in docs generation (eg. an ephemeral model, or a disabled model).dbt seed
has never been run).In either case, I think that failing the entire docs generation step for a single missing dataset is suboptimal. Can we either:
information_schemas
to only search in existing datasetsDatabaseError: 404 not found
, we can catch the error and print a warning. Note: running one query per dataset might incur a non-trivial cost on BigQuery.Steps To Reproduce
case #1: configure a model to build into a dataset that does not exist. Run
dbt docs generate
on BigQuery and observe the failure.case #2: configure a disabled model with:
run
dbt docs generate
on BigQuery and observe the failurecase #3: configure an ephemeral model with:
run
dbt docs generate
on BigQuery and observe the failureExpected behavior
dbt docs generate
should succeed even if a dataset present in the manifest does not exist in the database.System information
Which database are you using dbt with?
The output of
dbt --version
:The text was updated successfully, but these errors were encountered: