Incremental snapshot sql fails on postgres 9.6 #1665
Comments
drewbanin
changed the title
|
Thanks for this really thorough report @danielbcook! I was able to install postgres 9.6 locally, and I found that I see the exact same error message you do. Here's a minimal test case: Run it with: I was able to fix this by modifying the postgres implementation of It appears that the operative cast is in the lines: and Interestingly, we only need to cast the left side of the equality (eg. Last thing to say here: dbt is intended for use with analytical databases, and its core support for postgres is a function of dbt's Redshift support. While I'm happy to make this particular change (I think we can sneak it in for the imminent 0.14.1 release), we will generally be less aggressive in prioritizing bug fixes that target postgres, and especially those that effect older version of postgres! This particular issue doesn't appear to be a problem on pg >= 11, for instance. |
|
Yes! I’m two-for-two in valid bug reports. Thanks for considering this for a fix. Do you guys have a target date for 0.14.1?
On Aug 5, 2019, at 7:13 PM, Drew Banin <[email protected]<mailto:[email protected]>> wrote:
Thanks for this really thorough report @danielbcook<https:/danielbcook>! I was able to install postgres 9.6 locally, and I found that I see the exact same error message you do.
Here's a minimal test case:
…-- snapshots/my_snapshot.sql
{% snapshot my_snapshot %}
{{
config(
target_schema='archived_data',
updated_at='loaded_at',
strategy='timestamp',
unique_key='number')
}}
select
now() as loaded_at,
1 as number
{% endsnapshot %}
Run it with:
dbt snapshot # should succeed
dbt snapshot # should fail
I was able to fix this by modifying the postgres implementation of postgres__snapshot_merge_sql to look like:
{% macro postgres__snapshot_merge_sql(target, source, insert_cols) -%}
{%- set insert_cols_csv = insert_cols | join(', ') -%}
update {{ target }}
set dbt_valid_to = DBT_INTERNAL_SOURCE.dbt_valid_to
from {{ source }} as DBT_INTERNAL_SOURCE
where DBT_INTERNAL_SOURCE.dbt_scd_id = {{ target }}.dbt_scd_id
and DBT_INTERNAL_SOURCE.dbt_change_type::text = 'update'::text
and {{ target }}.dbt_valid_to is null;
insert into {{ target }} ({{ insert_cols_csv }})
select {% for column in insert_cols -%}
DBT_INTERNAL_SOURCE.{{ column }} {%- if not loop.last %}, {%- endif %}
{%- endfor %}
from {{ source }} as DBT_INTERNAL_SOURCE
where DBT_INTERNAL_SOURCE.dbt_change_type::text = 'insert'::text;
{% endmacro %}
It appears that the operative cast is in the lines:
and DBT_INTERNAL_SOURCE.dbt_change_type::text = 'update'::text
and
where DBT_INTERNAL_SOURCE.dbt_change_type::text = 'insert'::text;
Interestingly, we only need to cast the left side of the equality (eg. dbt_change_type::text). I totally expected the right side to be the problem! Let's cast them both for good measure.
Last thing to say here: dbt is intended for use with analytical databases, and its core support for postgres is a function of dbt's Redshift support. While I'm happy to make this particular change (I think we can sneak it in for the imminent 0.14.1 release), we will generally be less aggressive in prioritizing bug fixes that target postgres, and especially those that effect older version of postgres! This particular issue doesn't appear to be a problem on pg >= 11, for instance.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#1665>, or mute the thread<https:/notifications/unsubscribe-auth/AIUPWEUWBG5UD2MEONZEWYLQDDM35ANCNFSM4IJPUVHQ>.
|
|
Hoping to have an RC out tomorrow! We'll slip this in along with a couple of other in-progress changes -- should be live on all channels early next week! |
…-old-pg-versions (#1665) Fix for casting error on old versions of postgres in snapshots
Describe the bug
I set up a snapshot model and it runs OK on the first pass (where it creates the table), but on the subsequent passes it returns an error "failed to find conversion function from unknown to text". All of the fields in my Postgres query appear to be properly typed (none are 'unknown'). I suspect this is specific to UNION queries on Postgres 9.6.10 (and below).
Steps To Reproduce
Here is the original query (prior to many attempts to simplify and look for the culprit column). Simply run
dbt snapshottwice to reproduce. The results should return one row per day, per user, per table in the UNION which has a relevant row in it.Expected behavior
For subsequent runs of the snapshot to succeed like the first one does.
Screenshots and log output
If applicable, add screenshots or log output to help explain your problem.
System information
Which database are you using dbt with?
The output of
dbt --version:The operating system you're using:
The output of
python --version:Additional context
dbt.log output
The text was updated successfully, but these errors were encountered: