dbt compile should not need service account credentials

[ekhaydarov]

Describe the bug

When running dbt compile I get an error stating that GOOGLE_APPLIATION_CREDENTIALS is missing. compile builds run read models and sql statements. However, they dont actually need to be run. Therefore there is no need to have a functioning service account key in order to complete the compile step

Security wise it means you need to put the service account key into container to be able to run a simple compile. We shouldnt really be doing that?

Steps To Reproduce

Try running dbt compile without setting GOOGLE_APPLICATION_CREDENTIALS

Expected behavior

Be able to compile without getting an error that you need service account keys.

System information

Which database are you using dbt with?

• postgres
• redshift
• [ X bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

0.15.2

The operating system you're using: macos and ubuntu 18.04 and debian 9

The output of python --version: 3.7.4

Additional context

Add any other context about the problem here.

[beckjake]

This is expected behavior. The dbt compile command generates the sql that will be used in a create table as select ({{ sql }}) statement at runtime. Without having the ability to connect to the database to compile the sql, dbt would not be able to provide useful features like dbt_utils.get_relations_by_prefix

[drewbanin]

Agreed - this is working as intended. See #1584 for more information. I'm adding the wontfix label here as I don't anticipate making a code change in response to this issue, but i am more than happy to discuss it further :)

Can you tell me more about the dbt compile step you're performing? It sounds like this is separate from the dbt run step you're performing -- is this related to some sort of dockerization of your project?

[ekhaydarov]

Thanks for the quick response. My usage is basically the 1st point here. Visually inspecting the compiled output of model files. So essentially its a test and would like to perform it in CI. if it passes can just push the image to be used in k8s. in production the container will just mount the secret volume.

I think the need for credentials makes sense, so will just have to pass in and delete the secrets at the end. Not sure of a workaround to be honest, but will have to do.

Thanks

[ekhaydarov]

actually. slightly different. but same same. I have no run dbt compile successfully with credentials intact.

I now want to run dbt docs generate --no-compile. I would assume since all the assets have been compiled, doc generation would not need any credentials? yet i still get the error

09:37:00 | Building catalog
Encountered an error:
[Errno 2] No such file or directory: ...

Would it be possible to remove the need for credentials in this case since compilation is no longer required and assets have all the right things generated or is there another dependency that needs to have access to your data warehouse to generate docs

FYI dbt docs generate --no-compile doesnt even work on 0.15.2. seems fixed on 0.16.0

[drewbanin]

not so @ekhaydarov! The dbt docs generate does two things:

1. it executes dbt compile to build a manifest for your project
2. it queries the database to find the tables and columns (+types) in your database

The first step can be skipped with --no-compile, but the second step definitely requires database credentials to operate. I'd recommend running it in your production environment after your dbt run steps complete so that the generated docs reflect the actual state of the db after the run has occurred