Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ingestion/airflow-plugin): airflow remove old tasks #10485

Conversation

dushayntAW
Copy link
Contributor

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label May 10, 2024
@dushayntAW dushayntAW force-pushed the fix/ING-447/airflow-not-removing-old-task branch 2 times, most recently from ebd7762 to 49c689c Compare May 23, 2024 18:35
logger.debug("Initiating the cleanup of obsselete data from datahub")

ingested_dataflow_urns = list(
self.graph.get_urns_by_filter(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you should filter for cluster as well; otherwise if user has multiple Airflow instance you will delete dags which you shouldn't.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am filtering the entire URN which is already having the cluster i.e. urn:li:dataFlow:(airflow,simple_dag,prod)
So, still we need to match/filter cluster explicitly? or my understanding is wrong.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you check here we use cluster or env to generate the DataFlow Urns, so it is part of the urn. ->

self.urn = DataFlowUrn.create_from_ids(

This means if the env or cluster is set and has multiple Airflow environments like DEV and PROD, then your query will return the urns for both PROD and DEV, which we don't want in this case as these are different Airflow environment.

You should add cluster/env as a filter parameter.

airflow_job_urns: List = []

for dag in all_airflow_dags:
flow_urn = builder.make_data_flow_urn(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cluster should be passed in if exists

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as other comment

@dushayntAW dushayntAW requested a review from treff7es May 24, 2024 05:56
logger.debug("Initiating the cleanup of obsselete data from datahub")

ingested_dataflow_urns = list(
self.graph.get_urns_by_filter(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you check here we use cluster or env to generate the DataFlow Urns, so it is part of the urn. ->

self.urn = DataFlowUrn.create_from_ids(

This means if the env or cluster is set and has multiple Airflow environments like DEV and PROD, then your query will return the urns for both PROD and DEV, which we don't want in this case as these are different Airflow environment.

You should add cluster/env as a filter parameter.

@anshbansal anshbansal merged commit 177a50f into datahub-project:master Jun 10, 2024
54 checks passed
sleeperdeep pushed a commit to sleeperdeep/datahub that referenced this pull request Jun 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ingestion PR or Issue related to the ingestion of metadata
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants