Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Curious sortkey override behavior #2049

Closed
1 of 5 tasks
ryanaustincarlson opened this issue Jan 15, 2020 · 2 comments
Closed
1 of 5 tasks

Curious sortkey override behavior #2049

ryanaustincarlson opened this issue Jan 15, 2020 · 2 comments
Labels
bug Something isn't working redshift

Comments

@ryanaustincarlson
Copy link

ryanaustincarlson commented Jan 15, 2020

Describe the bug

We’ve run into an issue with defining sort config parameters in multiple places if both values are lists.

If…

  1. a default sort config is defined in the dbt_project;
  2. a model-specific sort parameter is defined; and
  3. both sort configurations are lists

then the resulting sort key is a concatenation of all of the keys, at least on Redshift (not sure how platform-dependent this might be).

Steps To Reproduce

Define a list of sort keys on a particular path in dbt_project

models:
  project_name:
    events:
      sort: ['occurred_at', 'client']

Then, create a new model in that path (e.g., project_name/events/cool_events_facts.sql) and redefine sort-keys

{{
    config(
        sort=['day_of_week','version']
    )
}}

When the model is built, the resulting SQL concatenates all four values, something like this (on Redshift):

create table
    events.cool_events_facts
    compound sortkey(day_of_week,version,occurred_at,client)
as (
    select *
    from whatever
);

Expected behavior

My expectation is that the model-specific sort param overrides those in dbt_project, which is what happens when a string literal is used instead of a list.

Screenshots and log output

LMK if you need any more info!

System information

Which database are you using dbt with?

  • postgres
  • redshift
  • bigquery
  • snowflake
  • other (specify: ____________)

The output of dbt --version:

installed version: 0.15.0
   latest version: 0.15.0

Up to date!

The operating system you're using:

macOS v10.14.6

The output of python --version:

Python 3.7.6

Additional context

Our current workaround is to define the model-specific sort params in dbt_project, which seems to override as expected.

@drewbanin
Copy link
Contributor

drewbanin commented Jan 20, 2020

Thanks for the report @ryanaustincarlson! I was able to reproduce this pretty easily, but it was tough getting to the bottom of the actual issue!

I started working on a fix for this over here: #2062. This PR should be shipped for our next patch release, v0.15.2 (or possibly 0.16.0).

Edit: This will be going out in the 0.16.0 Barbara Gittings release

@drewbanin
Copy link
Contributor

Fixed by #2062

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working redshift
Projects
None yet
Development

No branches or pull requests

2 participants