Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Table materialization is not using make_temp_relation or alias as identifier #2340

Closed
1 of 5 tasks
sumit0k opened this issue Apr 18, 2020 · 3 comments
Closed
1 of 5 tasks
Labels
bug Something isn't working good_first_issue Straightforward + self-contained changes, good for new contributors! stale Issues that have gone stale

Comments

@sumit0k
Copy link

sumit0k commented Apr 18, 2020

Describe the bug

Refering this particular file, it is present in new versions as well
https:/fishtown-analytics/dbt/blob/0.14.latest/core/dbt/include/global_project/macros/materializations/table/table.sql

The first 3 lines are, where for tmp and backup table, model['name'] is used instead of model['identifier']

  {%- set identifier = model['alias'] -%}
  {%- set tmp_identifier = model['name'] + '__dbt_tmp' -%}
  {%- set backup_identifier = model['name'] + '__dbt_backup' -%}

This results into failure while running the same model with different alias. Use usually do this to do backfilling, where our temporary tables are with alias this.table ~ '_backfill'. These model have same name as regular models but their aliases are different.

Steps To Reproduce

  1. Create a model with table materialization
  2. Create another project with same model name but with alias as this.table ~ '_backfill',. Sample config
{{
    config({
        "alias": this.table ~ '_backfill',
        "materialized": "table"
    })
}}
  1. Run both projects together.
  2. DBT would create same table in both runs with same name instead of using expected alias as prefix.

Expected behavior

Both run should finish successfully as their aliases are different and they are working on to create different table in database.

Proposed solution

make_temp_relation suffixes a timestamp with temp table name and that has solved this problem in our custom materializations. We are using something like

{%- set tmp_source_relation = make_temp_relation(target_relation, tmp_source_suffix) -%}

OR

Using this code to set the tmp_identifier would solve too. Although it is less resilient.

{%- set tmp_identifier = model['alias'] + '__dbt_tmp' -%}

Screenshots and log output

My model render failed with

2020-04-18 05:15:23,268 (Thread-6): On pm_transient___serverlogs_event: create  table
    "database_name"."schema_name"."pm_transient___serverlogs_event__dbt_tmp"
.......
2020-04-18 05:15:46,055 (Thread-6): Postgres error: duplicate key violates unique constraint "pg_class_relname_nsp_index" (possibly caused by concurrent transaction conflict)

2020-04-18 05:15:46,055 (Thread-6): On pm_transient___serverlogs_event: ROLLBACK

System information

Which database are you using dbt with?

  • postgres
  • redshift
  • bigquery
  • snowflake
  • other (specify: ____________)

The output of dbt --version:
I am running this in DBT Cloud (v0.14.4)

installed version: 0.14.4
   latest version: 0.16.1

Your version of dbt is out of date! You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation

The operating system you're using:
MacOS Catalina v10.15.4

The output of python --version:
Python 3.7.0

Additional context

Nope

@sumit0k sumit0k added bug Something isn't working triage labels Apr 18, 2020
@drewbanin drewbanin added good_first_issue Straightforward + self-contained changes, good for new contributors! and removed triage labels Apr 20, 2020
@drewbanin
Copy link
Contributor

Thanks for the report @sumitkumar1209!

Sure, I think using make_temp_relation is a good idea -- we use it in other materializations in dbt with a some success :)

The one thing to watch out for there is going to be #2197 -- we hit this bug in the make_temp_relation implementation on Postgres, and I think it might be possible to see the same behavior on Redshift too.

@sumit0k
Copy link
Author

sumit0k commented Apr 20, 2020

Yeah, understand that. That bug is anyway present for other materialization.

@github-actions
Copy link
Contributor

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

@github-actions github-actions bot added the stale Issues that have gone stale label Nov 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good_first_issue Straightforward + self-contained changes, good for new contributors! stale Issues that have gone stale
Projects
None yet
Development

No branches or pull requests

2 participants