Relationships tests do not run on 0.20.0 when -m model_name is used.

[jaypeedevlin]

Describe the bug

After upgrading to 0.20.0, relationships test don't run when specific models are selected— they act as if they don't exist.

• dbt ls --resource-type tests | grep relationships_ shows the tests
• dbt test runs all the test including the relationships
• dbt test -m model_with_relationship_tests runs all tests except relationships

I've reproduced this in three different projects inclduing a minimal jaffle_shop project. In both cases, downgrading to 0.19.1 had the tests run fine.

Steps To Reproduce

• Use dbt 0.20.0
• Find a model with a relationships test, or add one to a model
• dbt test -m model_name
• All tests with the model will be run except the relationship test(s)

Expected behavior

All tests to run, including relationships test

Screenshots and log output

System information

Which database are you using dbt with? (tested on more than one)

• postgres
• redshift
• bigquery
• snowflake
• other (specify: ____________)

The output of dbt --version:

installed version: 0.20.0
   latest version: 0.20.0

Up to date!

Plugins:
  - bigquery: 0.20.0
  - snowflake: 0.20.0
  - redshift: 0.20.0
  - postgres: 0.20.0

The operating system you're using: Mac

The output of python --version: 3.8.10

[jtcohen6]

Hey @jaypeedevlin, thanks for opening the issue—this has caused confusion for a few folks. I'll try to explain the decision behind this change. To start, I'm going to copy from my comment from #3496 (very similar):

This is intended as a feature in v0.20.0, implemented in #3235. (Original issue: #2891)

Test selection has always done something pretty subtle, which is the jump from "select a model" to "select all tests that directly depend on that model." We decided that dbt test -m model_name should be be a bit less greedy: it should select tests that depend model_name and only model_name, and which do not depend on any other resource—unless that other resource is also selected.

I've started using the term "indirect selection" to refer to that "jump." It's a powerful feature, since it means that dbt run -m my_model && dbt test -m my_model will do exactly what you'd expect—until you have a relationships test that fails because it's missing an upstream parent.

For that reason, dbt test -m stg_jaffle_shop__orders alone won't (indirectly) select the relationships test, since one of its parents is missing. On the other hand, of the following selection criteria will select that relationship test:

dbt test -m stg_jaffle_shop__orders stg_jaffle_shop__customers
dbt test -m stg_jaffle_shop__orders+1
dbt test -m stg_jaffle_shop__customers+1
dbt test -m @stg_jaffle_shop__orders

In the first case, the test is selected indirectly because both of its parents are selected. In the other three cases, the test is selected directly by a graph selection method.

There are docs for this updated behavior here, including the distinction between "direct" and "indirect" selection: https://docs.getdbt.com/reference/node-selection/test-selection-examples

One last piece of context from #3496 (comment):

We're working toward a dbt build command (run + test + seed + snapshot) for v0.21 (#2743). We wanted to make sure that, starting from a totally fresh schema, each of the following would succeed, and have results that make sense:

dbt build -s model_name
dbt build -s other_table_name
dbt build -s model_name other_table_name # run both models, unique test, equality test

Let's say the definition of other_table_name is just select * from {{ ref('model_name') }}. Then those would have the following behavior:

1. Run model_name + unique test
2. Run other_table_name
3. Run model_name, unique test, other_table_name, equality test (in that order, skipping downstream if anything fails)

I'm going to close this issue for the time being, but I'm curious to hear: Does this make sense to you? Do those docs help in clarifying the behavior?

[jaypeedevlin]

Hi Jeremy,

Thanks as always for the detailed response, and apologies for not checking either the release notes or closed 0.20.0rc issues before opening this one. Turns out this is one of those "feature not a bug" in the wild 😜

I can see the value in this change, particularly where Slim CI runs are concerned.

My initial concern on reading it was that it will make individual model development a pain, where you often want to run all tests, but knowing that model_name+ will select the dependent model solves that, it's just an update to the syntax when you're testing with relationship tests.

[bab354]

thanks @jtcohen6 that was really helpful!