Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add on_kill equivalent to Databricks SQL Hook to cancel timed out queries #42668

Open
wants to merge 208 commits into
base: main
Choose a base branch
from

Conversation

R7L208
Copy link
Contributor

@R7L208 R7L208 commented Oct 2, 2024

The Databricks Provider did not implement a mechanism to cancel SQL queries submitted by DatabricksSqlHook. This led to data quality issues, where Airflow would report a cancellation due to timeout; however, the corresponding SQL query would continue to run on Databricks.

This PR uses threading to cancel SQL queries submitted by DatabricksSqlHook.run() once the timeout is exceeded.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@R7L208 R7L208 force-pushed the lorin/databricks-sql-operator-on_kill-equivalent branch from 5a01d2a to 75f8c05 Compare October 2, 2024 17:36
@R7L208
Copy link
Contributor Author

R7L208 commented Oct 4, 2024

@Lee-W - Could you help me understand why the new exceptions are not being found in airflow/exceptions.py for jobs:

  • Tests / Provider checks / Compat 2.8.4:P3.8 provider check (pull_request)
  • Tests / Provider checks / Compat 2.9.3:P3.8 provider check (pull_request)
  • Tests / Provider checks / Compat 2.10.1:P3.8 provider check (pull_request)

The exceptions are present in the file and are able to be successfully imported locally when I run breeze testing tests --test-type "Providers[databricks]", so I'm struggling to understand why they would cause an import error here, specifically from /usr/local/lib/python3.8/site-packages/airflow/exceptions.py.

__ ERROR collecting tests/providers/databricks/sensors/test_databricks_sql.py __
ImportError while importing test module '/opt/airflow/tests/providers/databricks/sensors/test_databricks_sql.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/providers/databricks/sensors/test_databricks_sql.py:28: in <module>
    from airflow.providers.databricks.sensors.databricks_sql import DatabricksSqlSensor
/usr/local/lib/python3.8/site-packages/airflow/providers/databricks/sensors/databricks_sql.py:28: in <module>
    from airflow.providers.databricks.hooks.databricks_sql import DatabricksSqlHook
/usr/local/lib/python3.8/site-packages/airflow/providers/databricks/hooks/databricks_sql.py:41: in <module>
    from airflow.exceptions import (
E   ImportError: cannot import name 'AirflowTaskExecutionError' from 'airflow.exceptions' (/usr/local/lib/python3.8/site-packages/airflow/exceptions.py)
-- generated xml file: /files/test_result-providers_-amazon_google-sqlite.xml --

Also, the DB tests seem to be failing because of tests unrelated to my changes, or I'm missing how they are connected.

@potiuk
Copy link
Member

potiuk commented Oct 17, 2024

A LOT of conflicts. We moved providers to another directory, you need to resolve the conflicts/move your changes.

@R7L208 R7L208 requested a review from Lee-W October 17, 2024 23:34
@R7L208 R7L208 requested a review from Lee-W October 18, 2024 19:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.