[Regression] dbt 1.3 reads profiles.yml in current directory before ~/.dbt directory

[johnpkennedy]

Is this a regression in a recent version of dbt-core?

• I believe this is a regression in dbt-core functionality
• I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

I encountered the following error after upgrading from dbt-redshift 1.2.1 to 1.3

% dbt run
17:01:02  Encountered an error:
Parsing Error
  Env var required but not provided: 'DBT_REDSHIFT_USER'

As it turns out, dbt 1.3 is using the configuration from the profiles.yml file in my current directory instead of the profiles.yml in my ~/.dbt directory.

dbt debug --config-dir
17:06:39  Running with dbt=1.3.0
17:06:39  To view your profiles.yml file, run:

open /Users/<REDACTED>/Projects/<REDACTED>/dbt

We keep a separate profiles.yml in our dbt project directory for our CI/CD deployment workflow. This deployment profile does not have the user and pass properties set because we use environment variables.

Expected/Previous Behavior

dbt 1.2.1 (and earlier) reads the profiles.yml located in the ~/.dbt/ directory first.

% dbt debug --config-dir
15:15:41  Running with dbt=1.2.0
15:15:41  To view your profiles.yml file, run:

open /Users/<REDACTED>/.dbt

Steps To Reproduce

# Update brew to get the latest dbt recipes
brew update

# Upgrade dbt-redshift to 1.3
brew upgrade dbt-redshift

# Confirm the dbt version is 1.3
dbt --version

# Ensure that is a correct profiles.yml in your ~/.dbt directory

# Switch to your usual dbt project directory 
cd <project directory>

# Create another profiles.yml file in the current directory but omit the `user` and `pass` properties in the output. 

# Reproduce
dbt run

Relevant log output

No response

Environment

- OS: macOS 12.6
- Python: 3.10.8
- dbt (working version): 1.2.1
- dbt (regression version): 1.3

Which database adapter are you using with dbt?

redshift

Additional Context

No response

[jtcohen6]

@johnpkennedy Thanks for opening!

There's been a lot of conversation leading up to this change, including the design considerations involved: #5411, #5425

I don't believe this is a regression because:

• We didn't previously document that profiles.yml should be included as a root-level file in the dbt project. If anything, we really encouraged users not to do this.
• Previously, to run dbt using profiles.yml in the root directory, it was necessary to pass --profiles-dir=. or DBT_PROFILES_DIR=.. It should be equivalently easy to move the profile file into a subdirectory and pass it via flag / env var accordingly.

That said, I understand that this was a source of surprise for some folks. I think we should add a note about it to the "Breaking Changes" section of the v1.3 upgrade guide.

[wrb2]

We keep a separate profiles.yml in our dbt project directory for our CI/CD deployment workflow. This deployment profile does not have the user and pass properties set because we use environment variables.

We have the same setup (except for Redshift), and this change broke our workflow the same way.

I understand it is not a regression because it is documented, and looking at the previous discussions I see that lot of the points that I wanted to add (like this leading to people inevitably ending up with their password in git) were already mentioned.

Just advice on working around it for people whose workflow it also broke: don't try to use DBT_PROFILES_DIR or --profiles-dir= if you're also using sqlfluff, that won't work. Your best bet to hide the main profiles.yml into a subfolder or rename it to something dbt won't expect and then move or rename it in the CI job.

[mzayer-coveo]

@jtcohen6

The old way was working better for our case. It is indeed a breaking change that is not documented either. The v1.3 documentation still says that the default directory is ~/.dbt for profiles.

Is there a way to provide ~/.dbt to --profiles-dir?
When I do this, dbt looks for a downstream directory of ~/.dbt/

[johnpkennedy]

Hi @jtcohen6 - Respectively, the v1.2 documentation clearly states that the default directory is ~/.dbt.

https://docs.getdbt.com/docs/get-started/connection-profiles#related-documentation

[jtcohen6]

Sorry again to hear this one took you all by surprise. We anticipated this may affect a very small number of users.

@mzayer-coveo It sounds like you have a file named profiles.yml in the root directory where you run dbt, and you don't want to use it by default. Could you say more about why it's there?

It sounds like both @johnpkennedy and @wrborigin used that file to version-control a connection profile used in deployment, which is (hopefully) mostly a pass-through for environment variables. It should be just as easy to stick the profile in a folder and point there (via --profiles-dir) or add a cp deploy/profiles.yml . step to the deployment script / orchestration.

---

Is there a way to provide ~/.dbt to --profiles-dir?

This works for me locally:

$ dbt run --profiles-dir ~/.dbt

---

Respectively, the v1.2 documentation clearly states that the default directory is ~/.dbt.

We version our documentation, so that you're always reading docs correct for the version of dbt Core that you're using. Here's what you see when v1.3 is selected in the version dropdown up top:

As opposed to the previous behavior, when v1.2 is selected:

---

This change is called out in the changelog / release notes, and it's now also mentioned as a potentially breaking change for some users in the v1.3 upgrade guide:

[mzayer-coveo]

In our case, we were using ~/.dbt/ by default on our developer machines, and were using the profiles.yml in project root in the CI/CD pipeline we have by having --profiles-dir ./ in there. We fixed this by removing the --profiles-dir flag from CI/CD and adding DBT_PROFILES_DIR env var to our dev machines to point to ~/.dbt/ directory.