-
Notifications
You must be signed in to change notification settings - Fork 159
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create ExecutionMode.KUBERNETES example DAG & setup CI #535
Comments
@tatiana thanks you! Outdated documentation blocks POC using Cosmos.Looking forward to updated documentation |
Fix behaviour when using `ExecutionMode.KUBERNETES`, broken between the Cosmos releases 1.0.0 and 1.1.1. Update the documentation to be representative of the 1.x Cosmos interface: https://astronomer.github.io/astronomer-cosmos/getting_started/kubernetes.html Add unit tests to avoid regressions on these fixes. Part of the documentation fixes are made in: astronomer/cosmos-example#4 As a next step, we must ensure integration tests for running Cosmos on K8s to avoid this breaking change moving forward (issue #535). Closes: #493 Closes: #548 Closes: #534 Co-authored-by: Pádraic Slattery <[email protected]> (who created PR #551)
Fix behaviour when using `ExecutionMode.KUBERNETES`, broken between the Cosmos releases 1.0.0 and 1.1.1. Update the documentation to be representative of the 1.x Cosmos interface: https://astronomer.github.io/astronomer-cosmos/getting_started/kubernetes.html Add unit tests to avoid regressions on these fixes. Part of the documentation fixes are made in: astronomer/cosmos-example#4 As a next step, we must ensure integration tests for running Cosmos on K8s to avoid this breaking change moving forward (issue #535). Closes: #493 Closes: #548 Closes: #534 Co-authored-by: Pádraic Slattery <[email protected]>
An option could be to bring the Kubernetes example DAG to be part of the astronomer-cosmos repo and automate the steps we describe in our docs in Github Actions, using: https:/marketplace/actions/kubernetes-kind-cluster |
Fix behaviour when using `ExecutionMode.KUBERNETES`, broken between the Cosmos releases 1.0.0 and 1.1.1. Update the documentation to be representative of the 1.x Cosmos interface: https://astronomer.github.io/astronomer-cosmos/getting_started/kubernetes.html Add unit tests to avoid regressions on these fixes. Part of the documentation fixes are made in: astronomer/cosmos-example#4 As a next step, we must ensure integration tests for running Cosmos on K8s to avoid this breaking change moving forward (issue #535). Closes: #493 Closes: #548 Closes: #534 Co-authored-by: Pádraic Slattery <[email protected]>
Hi, @tatiana, I'm helping the Cosmos team manage their backlog and am marking this issue as stale. The issue involved creating an example DAG for ExecutionMode.KUBERNETES, adding it to a specified folder, referencing it in the documentation, setting up credentials for a K8s cluster for CI, and updating information for the CI to run the example DAG as part of integration tests. It seems that the issue has been resolved by bringing the Kubernetes example DAG to be part of the astronomer-cosmos repo and automating the steps described in the documentation using GitHub Actions. Could you please confirm if this issue is still relevant to the latest version of the Cosmos repository? If it is, please let the Cosmos team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days. Thank you for your understanding and cooperation. If you have any further questions or need assistance, feel free to reach out. |
This ticket is still relevant - since approximately 30% of Cosmos users use this mode |
It seems Github actions would allow us to spin up a KinD cluster - so we could automate running the DAG that we mention in our example (but is currently in a separate repo): |
I drafted a PR (#535) for this. The automation script works fine locally, but the Postgres instance is not healthy when running in the GitHub Action. Debug log:
|
To avoid our documentation becoming outdated in incompatible with the latest version of Cosmos, as described in #534, we should:
Create an example DAG, add it to the folder:
https:/astronomer/astronomer-cosmos/tree/main/dev/dags
Reference the new example DAG in the docs, using rst code block:
https:/astronomer/astronomer-cosmos/blob/main/docs/getting_started/execution-modes.rst?plain=1#L58-L61
https:/astronomer/astronomer-cosmos/blob/main/docs/getting_started/kubernetes.rst
Setup credentials to a K8s cluster that the CI can use to spin up the necessary resources
Update information so the CI runs this example DAG as part of the expensive integration tests:
astronomer-cosmos/pyproject.toml
Line 174 in bcf7714
astronomer-cosmos/pyproject.toml
Line 183 in bcf7714
The text was updated successfully, but these errors were encountered: