Fix DbtVirtualenvBaseOperator to use correct virtualenv Python path #1252
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Log Python binary info before command execution when using virtualenv - Reorder log statements for better readability and debugging
DAG pickling caused __init__-assigned methods to reference original instance state instead of newly created instances. Switch to properties for dynamic method assignment to ensure correct instance reference. Update tests to cover new behavior.
dosubot
bot
added
the
size:M
dosubot
bot
added
area:execution
✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR addresses an issue where the
DbtVirtualenvBaseOperatorwas executing dbt commands using the system-wide Python path instead of the virtualenv path. The root cause was that the self reference in therun_subprocessmethod was bound to a different instance than the one created during initialization, likely due to Airflow's DAG pickling mechanism.To resolve this, we've refactored the
invoke_dbtandhandle_exceptionmethods to be properties. This ensures that they dynamically reference the correct method of the current instance at runtime, rather than being bound to a potentially stale instance from initialization.Related Issue(s)
fix #1246
This may be related to issue #958 in version 1.5.0
Breaking Change?
No
Checklist
Additional Notes
I acknowledge that ideally, a test should be added to reproduce the original issue and verify the fix. However, I found it challenging to create an appropriate test, especially considering that this might require an integration test with Airflow to properly simulate Airflow behavior. If there are suggestions for how to effectively test this scenario, I would greatly appreciate the guidance.
I sincerely apologize for introducing this bug in the first place with PR #1200. I understand this has caused inconvenience, and I'm grateful for the opportunity to fix it. I kindly request a thorough review of these changes to ensure we've fully addressed the issue without introducing new problems.