Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Check Postgres relation name lengths and throw error when over 63 #2727

Merged
merged 4 commits into from
Aug 28, 2020
Merged

Check Postgres relation name lengths and throw error when over 63 #2727

merged 4 commits into from
Aug 28, 2020

Conversation

gshank
Copy link
Contributor

@gshank gshank commented Aug 26, 2020

resolves #2197

Description

Postgres table names (for both models and temporary tables) that are longer than 63 characters are silently truncated causing various problems in dbt processing.

Checklist

  • I have signed the CLA
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the CHANGELOG.md and added information about my change to the "dbt next" section.

@cla-bot
Copy link

cla-bot bot commented Aug 26, 2020

Thanks for your pull request, and welcome to our community! We require contributors to sign our Contributor License Agreement and we don't seem to have your signature on file. Check out this article for more information on why we have a CLA.

In order for us to review and merge your code, please submit the Individual Contributor License Agreement form attached above above. If you have questions about the CLA, or if you believe you've received this message in error, don't hesitate to ping @drewbanin.

CLA has not been signed by users: @gshank

@gshank gshank requested a review from beckjake August 26, 2020 15:14
@beckjake
Copy link
Contributor

@cla-bot check

@cla-bot cla-bot bot added the cla:yes label Aug 26, 2020
@cla-bot
Copy link

cla-bot bot commented Aug 26, 2020

The cla-bot has been summoned, and re-checked this pull request!

beckjake
beckjake suggested changes Aug 26, 2020
View reviewed changes
Copy link
Contributor

@beckjake beckjake left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good! I've added some feedback inline on particular things.

For the integration tests, this is my fault for not being clear, but we should try to not use the run a sql file and then test off it method if we can avoid it. Instead, let's use seed csvs and run dbt seed as part of the test suite. Check out the 052_column_quoting test as a good example of this pattern.

In the very long term, we'd like to move more tests into a common integration suite for all adapters (the dbt-adapter-tests suite), and .sql files are not very portable since every database can have its own silly syntax.

plugins/postgres/dbt/adapters/postgres/__init__.py Outdated Show resolved Hide resolved
plugins/postgres/dbt/adapters/postgres/relation.py Outdated Show resolved Hide resolved
plugins/redshift/dbt/adapters/redshift/__init__.py Outdated Show resolved Hide resolved
plugins/redshift/dbt/adapters/redshift/relation.py Outdated Show resolved Hide resolved
@gshank gshank requested a review from jtcohen6 August 26, 2020 18:22
jtcohen6
jtcohen6 reviewed Aug 27, 2020
View reviewed changes
Copy link
Contributor

@jtcohen6 jtcohen6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking great! I have one small comment about the error message. It looks like there are a few flake8 (code style) errors as well.

plugins/postgres/dbt/adapters/postgres/relation.py Outdated Show resolved Hide resolved
jtcohen6
jtcohen6 approved these changes Aug 27, 2020
View reviewed changes
Copy link
Contributor

@jtcohen6 jtcohen6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! I'll leave final code review to @beckjake

beckjake
beckjake approved these changes Aug 27, 2020
View reviewed changes
Copy link
Contributor

@beckjake beckjake left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! I have one tiny request on the changelog. Once that's done and tests pass, this is good to merge.

CHANGELOG.md
@@ -3,6 +3,7 @@

### Under the hood
- Added 3 more adapter methods that the new dbt-adapter-test suite can use for testing. ([#2492](https:/fishtown-analytics/dbt/issues/2492), [#2721](https:/fishtown-analytics/dbt/pull/2721))
- Check for Postgres relation names longer than 63 and throw exception. ([#2197](https:/fishtown-analytics/dbt/issues/2197))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should also add a link to this PR!

@gshank gshank merged commit 2cc2d97 into dev/marian-anderson Aug 28, 2020
@beckjake beckjake mentioned this pull request Sep 2, 2020
Merged
4 tasks
@kwigley kwigley deleted the fix/2197-long-table-names branch February 5, 2021 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@beckjake beckjake beckjake approved these changes

@jtcohen6 jtcohen6 jtcohen6 approved these changes

Assignees
No one assigned
Labels
cla:yes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Long table names can be truncated on Postgres (at least) silently
3 participants
@gshank @beckjake @jtcohen6