Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/operations actually #1328

Merged
merged 1 commit into from
Mar 1, 2019

Conversation

drewbanin
Copy link
Contributor

@drewbanin drewbanin commented Feb 28, 2019

Fixes #582, at least initially. This is a feature that we need to deploy and play around with to understand better, but it's difficult to use it for production workloads if it's not in a production release of dbt!

I suspect this functionality may change in meaningful ways, and I therefore suggest that:

  1. we discourage the use of the task for folks who aren't interested in bleeding-edge dbt features
  2. we do not document or promote this feature until (at least) the next release of dbt

I tried hiding the subtask from the argparse help menu, but i think argparse.SUPPRESS only works for flags (and not subtasks).

Once this is merged and we've had some time to experiment with it, we should queue up a subsequent issue to round out any rough edges and add some tasks.

Potential use cases:

  • vacuum/analyze on redshift
  • auto-compression for big tables on redshift
  • periodic unloads from redshift
  • copy into snowflake from s3
  • swap schemas (eg. for blue/green deployments)
  • really any out-of-band SQL that should be templated + versioned is suitable here

Really compellingly, we can supply entrypoint macros in packages, so the redshift package could contain macros that make it dead simple to do some really complex perf tuning!

Copy link
Member

@cmcarthur cmcarthur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks ok to me. i left you one little comment.

core/dbt/task/run_operation.py Outdated Show resolved Hide resolved
@drewbanin drewbanin merged commit 54b0b38 into dev/stephen-girard Mar 1, 2019
@drewbanin drewbanin deleted the feature/operations-actually branch March 1, 2019 16:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants