[CT-1813] [CT-1378] [Bug] Python models not picking up custom schema #393
Comments
github-actions
bot
changed the title
|
Found this after a quick search. Might be a bug on the connector side(although I am not too sure what is the connector version being used here in stored procedure). Will dig more when get to it. |
|
Good find, Chenyu! It sounds like this could be related to a bug in |
|
Will close out once we confirm that Snowflake has fixed this on their end |
|
bumping this from another community report: https://getdbt.slack.com/archives/C03QUA7DWCW/p1673461860389249 @jtcohen6 looks like the underlying Snowflake issues is fixed, could we fix here as well? |
|
+1 when/in what version can we expect this fix to get merged in? |
|
It looks like the linked issue/fix on Snowflake's side (snowflakedb/snowflake-connector-python#1274) will be included in a new version of We still need to test compatibility with that new version in |
github-actions
bot
changed the title
|
I'm getting the same error when running my project on dbt cloud. It works perfectly when building from the IDE but I get this error when I try running it in production. The SPs from the build are left in my production schema and I tried calling them manually from the snowflake UI and that works, if I've first selected the correct database and schema, maybe that it also a hint about what's going wrong? |
|
Seeing this error as well since moving to custom schemas. Any workarounds? Using dbt-snowflake: 1.3.0 Edit: |
|
Issue still persists with dbt-snowflake 1.4.1. Any update this issue? It was said here that this was to be fixed in 1.4.1. |
|
The (hopeful) fix for this was included in Since it's a major version bump, there's some additional due diligence warranted before including the upgrade in a v1.4.x patch release. |
|
Seems that the problem still exists in dbt-snowflake v1.4.1. |
|
Hey all, we're lining this up for our next sprint. So hopefully we should have something out in the next 2 weeks. |
|
Re-opening; this was closed due to automation. Once we've confirmed the changes in the backport we can close this. |
|
Here's an example breaking it
https://www.loom.com/share/a8771739cbfe401da1063be953f6e234
here is the log
[console_output.txt](https:/dbt-labs/dbt-snowflake/files/11052688/console_output.txt)
|
|
I feel like there's multiple things going on here. First, @Tonayya, @bama-troyel, the - name: dim_trial_data
description: dim_trial_data
config:
materialized: table
schema: dw
tags: ['python']Just covering the basis. An alternative is to put that in the Python model file itself: def model(dbt, session):
dbt.config(
materialized="table",
schema ="dw"
)Then @ernestoongaro, what happens if you have access to the base schema as well as the overridden one, and you don't drop anything during the run? ;) |
|
Is there any update on this issue or any workarounds? I am experiencing the same error as described in the opening and the last few most recent messages when running Should I expect this to be fixed? I'm using the following library versions: Please let me know if further information would be helpful |
|
I am also on: My custom schema is DEV_MODEL_python_model_test My python model: The error: I've tried to write the pandas_df to a snowpark df by running write_pandas e.g.: But then I get a new error which seems to have a syntax issue in the schema name (extra quotation marks): Oddly, when I try setting |
|
Hey @satpai1215 and @eaphilli, To cover the basics:
Starting from a new project ( Then add the following files in your project: def model(dbt, session):
dbt.config(
materialized="table",
schema="customSchemaSuffix"
)
df = dbt.ref("my_first_dbt_model")
return dfIn version: 2
models:
- name: my_python_model
description: my_python_model
config:
materialized: table
schema: customSchemaSuffix
tags: ['python']You can define the custom schema property in either the property file or the model file. Both should work. Do make sure that all of the following schemas exist and you have access to them (or you have enough privilege to have dbt create them on the fly):
|
|
Thank you for your response here! I can confirm that your base case scenario is working on my end. I've isolated the issue to problems occurring when returning a newly created dataframe from the Here is my test case below, that is returning the error that follows. Is there something I should be doing differently here? (apologies if this is not the correct forum for these types inquiries): Error Trace |
|
@patkearns10 and I had a chance to live-debug this with a user who's been running into the issue, and I think we got to the bottom of it! I'm still not sure why this is cropping up for only some users, and not others (me!) — and I'm pretty sure the root-cause fix here was (or should have been) in snowflakedb/snowflake-connector-python#1274 — but this managed to reliably solve the problem. The quickest fix was to add two lines within the Python model code, right before the def model(dbt, session):
dbt.config(schema="custom_schema")
pandas_df = dbt.ref("my_model").to_pandas()
# add these lines
session.use_database(dbt.this.database)
session.use_schema(dbt.this.schema)
return pandas_dfI believe we could just fix this for everyone by updating these lines to include those dbt-snowflake/dbt/include/snowflake/macros/materializations/table.sql Lines 49 to 51 in 0f77a08 Namely, by changing that to: if isinstance(df, pandas.core.frame.DataFrame):
session.use_database(target_relation.database)
session.use_schema(target_relation.schema)
# session.write_pandas does not have overwrite function
df = session.createDataFrame(df) |
|
Tracking this for our next sprint |
|
This issue also came up in community Slack: https://getdbt.slack.com/archives/CBSQTAPLG/p1683920034037219 |
|
I'm soon going to merge the fix Jeremy provided but admittedly on our system, as alluded to above, it's hard to recreate this. Even after this issue closes and the fix has been backported, I heartily encourage users that continue to find custom schema problems to speak up! |
|
I have just ran into this issue this week with switching our Python models to custom schemas. Is there an ETA or release version I can be looking for the fix to be released on? Thank you. |
|
Hey @waitsfornone, this should be out right now in 1.3.2, 1.4.3 and 1.5.1. |
Is this a new bug in dbt-core?
Current Behavior
When I try to create a python model using v1.3, I get an error that a schema was not provided. I have set a schema in my
dbt_project.ymlfile for the directory that the model is in, and have thegenerate_schema_nameset to change the schema based on the environment.Expected Behavior
The python model will run and create a model in the same schema as sql models in the same directory
Steps To Reproduce
dbt_project.ymlto a use a given schema, e.g.dbt/models/datamarts/dim/test_python.pyin that directory, e.g:generate_schema_name_for_envandgenerate_schema_name:Relevant log output
Environment
Which database adapter are you using with dbt?
snowflake
Additional Context
No response
The text was updated successfully, but these errors were encountered: