Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ExternalDeploymentSensor #1472

Merged
merged 5 commits into from
Feb 20, 2024
Merged

Conversation

pankajastro
Copy link
Contributor

@pankajastro pankajastro commented Feb 6, 2024

  • Introduce Astro Cloud connection type
  • Add ExternalDeploymentSensor, which uses Astro API to monitor Astro DAG status or task status.

Sensor expects external_dag_id as a required param and external_task_id as an optional param if external_task_id is provided then it monitors a task instance of task external_task_id otherwise monitors the overall status of external_dag_id.
Once the sensor execution starts it fetches the dag run for external_dag_id with status running or queued if found then waits for it to succeed or fail in the trigger component. if not found then return immediately and mark it a success.

TODO;

  • Test case
  • Docs/example
  • probably monitor task group too: This can't be done at moment because task group is just UI feature and not exposed externally

Assumption

When this sensor starts running it assumes that the dag it is monitoring is either in a running or queued state

@pankajastro pankajastro changed the title [WIP] Enhance external deployment task sesnor [WIP] Add ExternalDeploymentSensor Feb 6, 2024
@pankajastro pankajastro force-pushed the enhance_external_deployment_task_sesnor branch 6 times, most recently from df7d0c8 to cad8ba6 Compare February 7, 2024 04:24
Copy link

codecov bot commented Feb 7, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (e088bdd) 98.28% compared to head (ac900b1) 98.33%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1472      +/-   ##
==========================================
+ Coverage   98.28%   98.33%   +0.05%     
==========================================
  Files          91       94       +3     
  Lines        4485     4637     +152     
==========================================
+ Hits         4408     4560     +152     
  Misses         77       77              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@pankajastro pankajastro force-pushed the enhance_external_deployment_task_sesnor branch from 5e2e341 to b460f77 Compare February 7, 2024 05:51
@pankajastro pankajastro changed the title [WIP] Add ExternalDeploymentSensor Add ExternalDeploymentSensor Feb 7, 2024
@pankajastro
Copy link
Contributor Author

PR to fix CI: #1473

@pankajastro pankajastro marked this pull request as ready for review February 9, 2024 20:00
@pankajastro pankajastro force-pushed the enhance_external_deployment_task_sesnor branch 3 times, most recently from ea363e7 to 5ae9682 Compare February 12, 2024 08:25
astronomer/providers/core/example_dags/example_astro.py Outdated Show resolved Hide resolved
astronomer/providers/core/example_dags/example_astro.py Outdated Show resolved Hide resolved
astronomer/providers/core/example_dags/example_astro.py Outdated Show resolved Hide resolved
astronomer/providers/core/hooks/astro.py Outdated Show resolved Hide resolved
"""
base_url, _ = self.get_conn()
path = f"/api/v1/dags/{external_dag_id}/dagRuns"
params: dict[str, int | str | list[str]] = {"limit": 1, "state": ["running", "queued"]}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unsure why limit is set to 1 here?

Copy link
Contributor Author

@pankajastro pankajastro Feb 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just to fetch last dagrun based on execution date

astronomer/providers/core/sensors/astro.py Outdated Show resolved Hide resolved
astronomer/providers/core/sensors/astro.py Outdated Show resolved Hide resolved
astronomer/providers/core/sensors/astro.py Outdated Show resolved Hide resolved
state = dag_runs[0].get("state")
if state == "success":
return True
return False
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I understand correctly this fetches the latest DAG run and checks the status?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it check the status of last dagrun based on execution date

@pankajastro pankajastro force-pushed the enhance_external_deployment_task_sesnor branch from 7a83e83 to d578ccb Compare February 14, 2024 16:27
astronomer/providers/core/sensors/astro.py Outdated Show resolved Hide resolved
astronomer/providers/core/sensors/astro.py Outdated Show resolved Hide resolved
astronomer/providers/core/sensors/astro.py Outdated Show resolved Hide resolved
astronomer/providers/core/sensors/astro.py Outdated Show resolved Hide resolved
@phanikumv phanikumv force-pushed the enhance_external_deployment_task_sesnor branch from 9ffdcf9 to 0c41551 Compare February 16, 2024 06:09
@phanikumv phanikumv requested a review from BasPH February 16, 2024 07:03
@pankajastro pankajastro force-pushed the enhance_external_deployment_task_sesnor branch from 0c41551 to ac900b1 Compare February 19, 2024 09:04
@pankajastro pankajastro merged commit 3ca9989 into main Feb 20, 2024
11 checks passed
@pankajastro pankajastro deleted the enhance_external_deployment_task_sesnor branch February 20, 2024 11:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants