Store compiled_sql even when task fails (fixes issue #369) #671
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 3 commits
November 14, 2023 09:39
…ered by addition of full_refresh to template fields
👷 Deploy Preview for amazing-pothos-a3bca0 processing.
|
dosubot
bot
added
area:execution
|
I wasn't sure if additional unit tests are needed for this. I can add them if so. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #671 +/- ##
=======================================
Coverage 92.75% 92.75%
=======================================
Files 54 54
Lines 2235 2235
=======================================
Hits 2073 2073
Misses 162 162 ☔ View full report in Codecov by Sentry. |
tatiana
approved these changes
Nov 14, 2023
tatiana
pushed a commit
that referenced
this pull request
Nov 15, 2023
Update `DbtLocalBaseOperator` code to store `compiled_sql` prior to exception handling so that when a task fails, the `compiled_sql` can still be reviewed. In the process found and fixed a related bug where `compiled_sql` was being dropped on some operations due to the way that the `full_refresh` field was being added to the `template_fields`. Closes #369 Fixes bug introduced in #623 where compiled_sql was being lost in `DbtSeedLocalOperator` and `DbtRunLocalOperator` Co-authored-by: Andrew Greenburg <[email protected]> (cherry picked from commit ee91ece)
tatiana
added a commit
that referenced
this pull request
Nov 15, 2023
Bug fixes * Store `compiled_sql` even when task fails by @agreenburg in #671 * Refactor `LoadMethod.LOCAL` to use symlinks instead of copying directory by @jbandoro in #660 * Fix 'Unable to find the dbt executable: dbt' error by @tatiana in #666 * Fix installing deps when using `profile_mapping` & `ExecutionMode.LOCAL` by @joppevos in #659 Others * Docs fix: add execution config to MWAA code example by @ugmuka in #674
Merged
tatiana
added a commit
that referenced
this pull request
Nov 15, 2023
Bug fixes * Store `compiled_sql` even when task fails by @agreenburg in #671 * Refactor `LoadMethod.LOCAL` to use symlinks instead of copying directory by @jbandoro in #660 * Fix 'Unable to find the dbt executable: dbt' error by @tatiana in #666 * Fix installing deps when using `profile_mapping` & `ExecutionMode.LOCAL` by @joppevos in #659 Others * Docs fix: add execution config to MWAA code example by @ugmuka in #674 (cherry picked from commit aa9b7bb)
Merged
tatiana
pushed a commit
that referenced
this pull request
Nov 15, 2023
Update `DbtLocalBaseOperator` code to store `compiled_sql` prior to exception handling so that when a task fails, the `compiled_sql` can still be reviewed. In the process found and fixed a related bug where `compiled_sql` was being dropped on some operations due to the way that the `full_refresh` field was being added to the `template_fields`. Closes #369 Fixes bug introduced in #623 where compiled_sql was being lost in `DbtSeedLocalOperator` and `DbtRunLocalOperator` Co-authored-by: Andrew Greenburg <[email protected]> (cherry picked from commit ee91ece)
tatiana
added a commit
that referenced
this pull request
Nov 15, 2023
Bug fixes * Store `compiled_sql` even when task fails by @agreenburg in #671 * Refactor `LoadMethod.LOCAL` to use symlinks instead of copying directory by @jbandoro in #660 * Fix 'Unable to find the dbt executable: dbt' error by @tatiana in #666 * Fix installing deps when using `profile_mapping` & `ExecutionMode.LOCAL` by @joppevos in #659 Others * Docs fix: add execution config to MWAA code example by @ugmuka in #674
|
Thanks for the contribution, @agreenburg , we released it as part of Cosmos 1.2.4 yesterday: |
tatiana
added a commit
that referenced
this pull request
Nov 16, 2023
**Bug fixes** * Store `compiled_sql` even when task fails by @agreenburg in #671 * Refactor `LoadMethod.LOCAL` to use symlinks instead of copying directory by @jbandoro in #660 * Fix 'Unable to find the dbt executable: dbt' error by @tatiana in #666 * Fix installing deps when using `profile_mapping` & `ExecutionMode.LOCAL` by @joppevos in #659 **Others** * Docs fix: add execution config to MWAA code example by @ugmuka in #674 (cherry picked from commit aa9b7bb)
arojasb3
pushed a commit
to arojasb3/astronomer-cosmos
that referenced
this pull request
Jul 14, 2024
…stronomer#671) Update `DbtLocalBaseOperator` code to store `compiled_sql` prior to exception handling so that when a task fails, the `compiled_sql` can still be reviewed. In the process found and fixed a related bug where `compiled_sql` was being dropped on some operations due to the way that the `full_refresh` field was being added to the `template_fields`. Closes astronomer#369 Fixes bug introduced in astronomer#623 where compiled_sql was being lost in `DbtSeedLocalOperator` and `DbtRunLocalOperator` Co-authored-by: Andrew Greenburg <[email protected]>
arojasb3
pushed a commit
to arojasb3/astronomer-cosmos
that referenced
this pull request
Jul 14, 2024
**Bug fixes** * Store `compiled_sql` even when task fails by @agreenburg in astronomer#671 * Refactor `LoadMethod.LOCAL` to use symlinks instead of copying directory by @jbandoro in astronomer#660 * Fix 'Unable to find the dbt executable: dbt' error by @tatiana in astronomer#666 * Fix installing deps when using `profile_mapping` & `ExecutionMode.LOCAL` by @joppevos in astronomer#659 **Others** * Docs fix: add execution config to MWAA code example by @ugmuka in astronomer#674 (cherry picked from commit aa9b7bb)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Updated DbtLocalBaseOperator code to store compiled_sql prior to exception handling so that when a task fails the compiled_sql can still be reviewed.
In the process found and fixed a related bug where compiled_sql was being dropped on some operations due to the way that the
full_refreshfield was being added to the template_fields.Related Issue(s)
closes #369
Fixes bug introduced in #623 where compiled_sql was being lost in
DbtSeedLocalOperatorandDbtRunLocalOperatorBreaking Change?
Not a breaking change
Checklist