-
Notifications
You must be signed in to change notification settings - Fork 159
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add cosmos/propagate_logs Airflow config support for disabling log pr… #648
Add cosmos/propagate_logs Airflow config support for disabling log pr… #648
Conversation
…opagation if desired
👷 Deploy Preview for amazing-pothos-a3bca0 processing.
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #648 +/- ##
==========================================
+ Coverage 92.67% 92.70% +0.02%
==========================================
Files 54 54
Lines 2212 2220 +8
==========================================
+ Hits 2050 2058 +8
Misses 162 162
☔ View full report in Codecov by Sentry. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you very much @agreenburg for making this improvement to Cosmos logging.
Initially I hadn't thought about having Cosmos as a provider, but it can make sense. It fits Airflow's definition of a provider: https://airflow.apache.org/docs/apache-airflow-providers/
Could you give more details on the following?
"Expose entry-point so that Airflow can theoretically detect configuration default."
I'm happy to merge this change once I understand this a little bit more :)
HI @tatiana ! The entry-point stuff is based on the guidance here: https://airflow.apache.org/docs/apache-airflow-providers/howto/create-custom-providers.html#how-to-create-a-provider I have not thoroughly tested that airflow is detecting it properly yet though. |
Ah, I see - so the motivation is to display the config in the Airflow UI (e.g. http://localhost:8080/configuration). If we didn't add this provider logic (both entry point and get_provider_info), you'd still be able to set this value via an environment variable or Airflow config file, but it would not display in the UI. Makes sense! |
…opagation if desired (#648) Add Airflow config check for cosmos/propagate_logs to allow override of default propagation behavior. Expose entry-point so that Airflow can theoretically detect configuration default. Closes #639 ## Breaking Change? This is backward-compatible as it falls back to default behavior if the `cosmos` section or `propagate_logs` option don't exist. ## Checklist - [X] I have made corresponding changes to the documentation (if required) - [X] I have added tests that prove my fix is effective or that my feature works --------- Co-authored-by: Andrew Greenburg <[email protected]>
Features * Add ProfileMapping for Vertica by @perttus in #540 and #688 * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649 * Add support to select using (some) graph operators when using LoadMode.CUSTOM and LoadMode.DBT_MANIFEST by @tatiana in #728 * Add cosmos/propagate_logs Airflow config support for disabling log pr… by @agreenburg in #648 * Add operator_args full_refresh as a templated field by @joppevos in #623 * Expose environment variables and dbt variables in ProjectConfig by @jbandoro in #735 Enhancements * Make Pydantic an optional dependency by @pixie79 in #736 * Create a symbolic link to dbt_packages when dbt_deps is False when using LoadMode.DBT_LS by @DanMawdsleyBA in #730 * Support no profile_config for ExecutionMode.KUBERNETES and ExecutionMode.DOCKER by @MrBones757 and @tatiana in #681 and #731 * Add aws_session_token for Athena mapping by @benjamin-awd in #663 Others * Replace flake8 for Ruff by @joppevos in #743 * Reduce code complexity to 8 by @joppevos in #738 * Update conflict matrix between Airflow and dbt versions by @tatiana in #731 * Speed up integration tests by @jbandoro in #732
Features * Add ProfileMapping for Vertica by @perttus in #540 and #688 * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649 * Add support to select using (some) graph operators when using LoadMode.CUSTOM and LoadMode.DBT_MANIFEST by @tatiana in #728 * Add cosmos/propagate_logs Airflow config support for disabling log pr… by @agreenburg in #648 * Add operator_args full_refresh as a templated field by @joppevos in #623 * Expose environment variables and dbt variables in ProjectConfig by @jbandoro in #735 Enhancements * Make Pydantic an optional dependency by @pixie79 in #736 * Create a symbolic link to dbt_packages when dbt_deps is False when using LoadMode.DBT_LS by @DanMawdsleyBA in #730 * Support no profile_config for ExecutionMode.KUBERNETES and ExecutionMode.DOCKER by @MrBones757 and @tatiana in #681 and #731 * Add aws_session_token for Athena mapping by @benjamin-awd in #663 Others * Replace flake8 for Ruff by @joppevos in #743 * Reduce code complexity to 8 by @joppevos in #738 * Update conflict matrix between Airflow and dbt versions by @tatiana in #731 * Speed up integration tests by @jbandoro in #732
**Features** * Add `ProfileMapping` for Snowflake encrypted private key path by @ivanstillfront in #608 * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649 * Add `DbtDocsGCSOperator` for uploading dbt docs to GCS by @jbandoro in #616 * Add support to select using (some) graph operators when using `LoadMode.CUSTOM` and `LoadMode.DBT_MANIFEST` by @tatiana in #728 * Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in #648 * Add operator_args ``full_refresh`` as a templated field by @joppevos in #623 * Expose environment variables and dbt variables in ``ProjectConfig`` by @jbandoro in #735 **Enhancements** * Make Pydantic an optional dependency by @pixie79 in #736 * Create a symbolic link to `dbt_packages` when `dbt_deps` is False when using `LoadMode.DBT_LS` by @DanMawdsleyBA in #730 * Support no `profile_config` for `ExecutionMode.KUBERNETES` and `ExecutionMode.DOCKER` by @MrBones757 and @tatiana in #681 and #731 * Add `aws_session_token` for Athena mapping by @benjamin-awd in #663 **Others** * Replace flake8 for Ruff by @joppevos in #743 * Reduce code complexity to 8 by @joppevos in #738 * Update conflict matrix between Airflow and dbt versions by @tatiana in #731 * Speed up integration tests by @jbandoro in #732
**Features** * Add new parsing method ``LoadMode.DBT_LS_FILE`` by @woogakoki in #733 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/parsing-methods.html#dbt-ls-file)). * Add support to select using (some) graph operators when using ``LoadMode.CUSTOM`` and ``LoadMode.DBT_MANIFEST`` by @tatiana in #728 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/selecting-excluding.html#using-select-and-exclude)) * Add support for dbt ``selector`` arg for DAG parsing by @jbandoro in #755, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#render-config)). * Add ``ProfileMapping`` for Vertica by @perttus in #540, #688 and #741, as ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/VerticaUserPassword.html)). * Add ``ProfileMapping`` for Snowflake encrypted private key path by @ivanstillfront in #608, as ([documentation]( https://astronomer.github.io/astronomer-cosmos/profiles/SnowflakeEncryptedPrivateKeyFilePem.html)). * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in #649 * Add ``DbtDocsGCSOperator`` for uploading dbt docs to GCS by @jbandoro in #616, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/generating-docs.html#upload-to-gcs)). * Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in #648, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/logging.html)). * Add operator_args ``full_refresh`` as a templated field by @joppevos in #623 * Expose environment variables and dbt variables in ``ProjectConfig`` by @jbandoro in #735 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/project-config.html#project-config-example)). * Support disabling event tracking when using Cosmos profile mapping by @jbandoro in #768, ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/index.html#disabling-dbt-event-tracking)). **Enhancements** * Make Pydantic an optional dependency by @pixie79 in #736 * Create a symbolic link to ``dbt_packages`` when ``dbt_deps`` is False when using ``LoadMode.DBT_LS`` by @DanMawdsleyBA in #730 * Add ``aws_session_token`` for Athena mapping by @benjamin-awd in #663 * Retrieve temporary credentials from ``conn_id`` for Athena by @octiva in #758 * Extend ``DbtDocsLocalOperator`` with static flag by @joppevos in #759 **Bug fixes** * Remove Pydantic upper version restriction so Cosmos can be used with Airflow 2.8 by @jlaneve in #772 **Others** * Replace flake8 for Ruff by @joppevos in #743 * Reduce code complexity to 8 by @joppevos in #738 * Speed up integration tests by @jbandoro in #732 * Fix README quickstart link in by @RNHTTR in #776 * Add package location to work with hatchling 1.19.0 by @jbandoro in #761 * Fix type check error in ``DbtKubernetesBaseOperator.build_env_args`` by @jbandoro in #766 * Improve ``DBT_MANIFEST`` documentation by @dwreeves in #757 * Update conflict matrix between Airflow and dbt versions by @tatiana in #731 and #779 * pre-commit updates in #775, #770, #762
**Features** * Add new parsing method ``LoadMode.DBT_LS_FILE`` by @woogakoki in astronomer#733 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/parsing-methods.html#dbt-ls-file)). * Add support to select using (some) graph operators when using ``LoadMode.CUSTOM`` and ``LoadMode.DBT_MANIFEST`` by @tatiana in astronomer#728 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/selecting-excluding.html#using-select-and-exclude)) * Add support for dbt ``selector`` arg for DAG parsing by @jbandoro in astronomer#755, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#render-config)). * Add ``ProfileMapping`` for Vertica by @perttus in astronomer#540, astronomer#688 and astronomer#741, as ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/VerticaUserPassword.html)). * Add ``ProfileMapping`` for Snowflake encrypted private key path by @ivanstillfront in astronomer#608, as ([documentation]( https://astronomer.github.io/astronomer-cosmos/profiles/SnowflakeEncryptedPrivateKeyFilePem.html)). * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in astronomer#649 * Add ``DbtDocsGCSOperator`` for uploading dbt docs to GCS by @jbandoro in astronomer#616, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/generating-docs.html#upload-to-gcs)). * Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in astronomer#648, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/logging.html)). * Add operator_args ``full_refresh`` as a templated field by @joppevos in astronomer#623 * Expose environment variables and dbt variables in ``ProjectConfig`` by @jbandoro in astronomer#735 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/project-config.html#project-config-example)). * Support disabling event tracking when using Cosmos profile mapping by @jbandoro in astronomer#768, ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/index.html#disabling-dbt-event-tracking)). **Enhancements** * Make Pydantic an optional dependency by @pixie79 in astronomer#736 * Create a symbolic link to ``dbt_packages`` when ``dbt_deps`` is False when using ``LoadMode.DBT_LS`` by @DanMawdsleyBA in astronomer#730 * Add ``aws_session_token`` for Athena mapping by @benjamin-awd in astronomer#663 * Retrieve temporary credentials from ``conn_id`` for Athena by @octiva in astronomer#758 * Extend ``DbtDocsLocalOperator`` with static flag by @joppevos in astronomer#759 **Bug fixes** * Remove Pydantic upper version restriction so Cosmos can be used with Airflow 2.8 by @jlaneve in astronomer#772 **Others** * Replace flake8 for Ruff by @joppevos in astronomer#743 * Reduce code complexity to 8 by @joppevos in astronomer#738 * Speed up integration tests by @jbandoro in astronomer#732 * Fix README quickstart link in by @RNHTTR in astronomer#776 * Add package location to work with hatchling 1.19.0 by @jbandoro in astronomer#761 * Fix type check error in ``DbtKubernetesBaseOperator.build_env_args`` by @jbandoro in astronomer#766 * Improve ``DBT_MANIFEST`` documentation by @dwreeves in astronomer#757 * Update conflict matrix between Airflow and dbt versions by @tatiana in astronomer#731 and astronomer#779 * pre-commit updates in astronomer#775, astronomer#770, astronomer#762
@agreenburg Have you tested if the config resolved the double logging issue? I found airflow is not taking this as provider so the config doesn't really works for me |
I was able to test it with both an environment variable and by adding to the airflow.cfg and it was suppressing the propagation as expected. Not sure what's missing to get it to work as a provider. |
Got it thanks. I used the following command to check airflow config and it looks good to me (can get False as return)
But actual log still shows doubled
|
Those are two separate log messages. One with the Inlets[] and the other with the Outlets[]. Before the change I was seeing logs like this:
|
@agreenburg Thanks for the explanation. I guess I misunderstood the outcome with the setting.
and completely get rid of the cosmos log
as it's not that readable and format well... |
@zy-wiser yes, unfortunately that would require a bigger change. I guess the custom logging wasn't designed with the celery executor in mind. |
…opagation if desired (astronomer#648) Add Airflow config check for cosmos/propagate_logs to allow override of default propagation behavior. Expose entry-point so that Airflow can theoretically detect configuration default. Closes astronomer#639 ## Breaking Change? This is backward-compatible as it falls back to default behavior if the `cosmos` section or `propagate_logs` option don't exist. ## Checklist - [X] I have made corresponding changes to the documentation (if required) - [X] I have added tests that prove my fix is effective or that my feature works --------- Co-authored-by: Andrew Greenburg <[email protected]>
**Features** * Add `ProfileMapping` for Snowflake encrypted private key path by @ivanstillfront in astronomer#608 * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in astronomer#649 * Add `DbtDocsGCSOperator` for uploading dbt docs to GCS by @jbandoro in astronomer#616 * Add support to select using (some) graph operators when using `LoadMode.CUSTOM` and `LoadMode.DBT_MANIFEST` by @tatiana in astronomer#728 * Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in astronomer#648 * Add operator_args ``full_refresh`` as a templated field by @joppevos in astronomer#623 * Expose environment variables and dbt variables in ``ProjectConfig`` by @jbandoro in astronomer#735 **Enhancements** * Make Pydantic an optional dependency by @pixie79 in astronomer#736 * Create a symbolic link to `dbt_packages` when `dbt_deps` is False when using `LoadMode.DBT_LS` by @DanMawdsleyBA in astronomer#730 * Support no `profile_config` for `ExecutionMode.KUBERNETES` and `ExecutionMode.DOCKER` by @MrBones757 and @tatiana in astronomer#681 and astronomer#731 * Add `aws_session_token` for Athena mapping by @benjamin-awd in astronomer#663 **Others** * Replace flake8 for Ruff by @joppevos in astronomer#743 * Reduce code complexity to 8 by @joppevos in astronomer#738 * Update conflict matrix between Airflow and dbt versions by @tatiana in astronomer#731 * Speed up integration tests by @jbandoro in astronomer#732
**Features** * Add new parsing method ``LoadMode.DBT_LS_FILE`` by @woogakoki in astronomer#733 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/parsing-methods.html#dbt-ls-file)). * Add support to select using (some) graph operators when using ``LoadMode.CUSTOM`` and ``LoadMode.DBT_MANIFEST`` by @tatiana in astronomer#728 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/selecting-excluding.html#using-select-and-exclude)) * Add support for dbt ``selector`` arg for DAG parsing by @jbandoro in astronomer#755, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/render-config.html#render-config)). * Add ``ProfileMapping`` for Vertica by @perttus in astronomer#540, astronomer#688 and astronomer#741, as ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/VerticaUserPassword.html)). * Add ``ProfileMapping`` for Snowflake encrypted private key path by @ivanstillfront in astronomer#608, as ([documentation]( https://astronomer.github.io/astronomer-cosmos/profiles/SnowflakeEncryptedPrivateKeyFilePem.html)). * Add support for Snowflake encrypted private key environment variable by @DanMawdsleyBA in astronomer#649 * Add ``DbtDocsGCSOperator`` for uploading dbt docs to GCS by @jbandoro in astronomer#616, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/generating-docs.html#upload-to-gcs)). * Add cosmos/propagate_logs Airflow config support for disabling log propagation by @agreenburg in astronomer#648, ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/logging.html)). * Add operator_args ``full_refresh`` as a templated field by @joppevos in astronomer#623 * Expose environment variables and dbt variables in ``ProjectConfig`` by @jbandoro in astronomer#735 ([documentation](https://astronomer.github.io/astronomer-cosmos/configuration/project-config.html#project-config-example)). * Support disabling event tracking when using Cosmos profile mapping by @jbandoro in astronomer#768, ([documentation](https://astronomer.github.io/astronomer-cosmos/profiles/index.html#disabling-dbt-event-tracking)). **Enhancements** * Make Pydantic an optional dependency by @pixie79 in astronomer#736 * Create a symbolic link to ``dbt_packages`` when ``dbt_deps`` is False when using ``LoadMode.DBT_LS`` by @DanMawdsleyBA in astronomer#730 * Add ``aws_session_token`` for Athena mapping by @benjamin-awd in astronomer#663 * Retrieve temporary credentials from ``conn_id`` for Athena by @octiva in astronomer#758 * Extend ``DbtDocsLocalOperator`` with static flag by @joppevos in astronomer#759 **Bug fixes** * Remove Pydantic upper version restriction so Cosmos can be used with Airflow 2.8 by @jlaneve in astronomer#772 **Others** * Replace flake8 for Ruff by @joppevos in astronomer#743 * Reduce code complexity to 8 by @joppevos in astronomer#738 * Speed up integration tests by @jbandoro in astronomer#732 * Fix README quickstart link in by @RNHTTR in astronomer#776 * Add package location to work with hatchling 1.19.0 by @jbandoro in astronomer#761 * Fix type check error in ``DbtKubernetesBaseOperator.build_env_args`` by @jbandoro in astronomer#766 * Improve ``DBT_MANIFEST`` documentation by @dwreeves in astronomer#757 * Update conflict matrix between Airflow and dbt versions by @tatiana in astronomer#731 and astronomer#779 * pre-commit updates in astronomer#775, astronomer#770, astronomer#762
…opagation if desired
Description
Add Airflow config check for cosmos/propagate_logs to allow override of default propagation behavior.
Expose entry-point so that Airflow can theoretically detect configuration default.
Related Issue(s)
closes #639
Breaking Change?
This is backward-compatible as it falls back to default behavior if the
cosmos
section orpropagate_logs
option don't exist.Checklist