…ks_sql

This reverts commit 13287fcc63cfe2f3df548c287bd6386007520167.

…ok.run

…lent

Co-authored-by: Wei Lee <[email protected]>

…lent

…lent

Co-authored-by: Wei Lee <[email protected]>

Just simplified redundant code from apache#42737

- Added a unit test

…che#42629)

* Include object router and migrate objects/historical_metrics_data to FastAPI

* Use pyfixture session

* Include provide_session for SQLite session thread issue

* Make method async and remove unused session from tests

* Include return type to generate proper openapi spec

* Object definition to Dashboard definition, make safe date Annotated parameter in parameters.py

* Include pydantic model for Dashboard Historical Metric Data and convert response to HistoricalMetricDataResponse

* Fix conflict and rerun pre-commit hooks

* Rename data variable to historical metrics for consistency

* Include object router and migrate objects/historical_metrics_data to FastAPI

* Object definition to Dashboard definition, make safe date Annotated parameter in parameters.py

* Drop data from method name to prevent double Data in the name of objects

* Variable name change to prevent shadow naming with method name

* Resolve conflicts and rebase again

* Resolve conflicts and rebase

* Add support to filter by last dagrun state in UI.

* Fix lint errors.

* Fix lint errors.

* Fix PR comments over null checks and query parameter to be last_dag_run_state.

…pache#41814)

* feat(datasets): make strict_dataset_uri_validation default to True

* docs(assets): add uri scheme restrict instruction for AIP-60

* feat(assets): add an error logging to URI that is not AIP-60 compliant

* Better test string

---------

Co-authored-by: Tzu-ping Chung <[email protected]>

* fix: always defer once more after log fetching to ensure pod completion is handled

* add tests

* removing print

---------

Co-authored-by: Jean-Eudes Peloye <[email protected]>

)

closes apache#39584

This PR adds templating information related to Python operators in How-to Guide

Handle redacted value for Variables

Previously this lived in test_scheduler_job.py

It only really tested the behavior of DagRun.update_state.

As far as I can tell, it checks that if you null out the state on a TI of a finished dag, and then you call ``update_state``, then the DR will be set to running.

…pache#42830)" (apache#42864)

This reverts commit 3b4c73a.

* Prepare docs for Oct 1st adhoc wave of providers

* Update

* update amazon

This PR reverts Core Extension doc change (https://airflow.apache.org/docs/apache-airflow-providers/core-extensions/index.html) from apache#41348

Since those docs only work on Airflow stable, we can only change this after 3.0

…roject (apache#42505)

This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.

In addition to the straight file rename the other changes I had to make here
are:

- Some mypy/typing fixes.

  Mypy can be fragile about what it picks up when, so maybe some of those
  changes were caused by that. But the typing changes aren't large.

- Improve typing in common.sql type stub

  Again, likely a mypy file oddity, but the types should be safe

- Removed the `check-providers-init-file-missing` check

  This isn't needed now that airflow/providers shouldn't exist at all in the
  main tree.

- Create a "dev.tests_common" package that contains helper files and common
  pytest fixtures

  Since the provider tests are no longer under tests/ they don't automatically
  share the fixtures from the parent `tests/conftest.py` so they needed
  extracted.

  Ditto for `tests.test_utils` -- they can't be easily imported in provider
  tests anymore, so they are moved to a more explicit shared location.

In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.

)

* Improving validation of task retries to handle None values

* Updated the validation check to use the "or" logical operator

Co-authored-by: Jens Scheffler <[email protected]>

* Added a comment for the new validation check

* Update baseoperator.py

* Update taskinstance.py

---------

Co-authored-by: Jens Scheffler <[email protected]>

Before
```
❯ pre-commit install
[WARNING] top-level `default_stages` uses deprecated stage names (commit, push) which will be removed in a future version.  run: `pre-commit migrate-config` to automatically fix this.
pre-commit installed at .git/hooks/pre-commit
```

…thods (apache#42871)

Bump `blacken-docs` pre-commit from 1.18.0 -> 1.19.0

This fixes the issue we saw in https://apache-airflow.slack.com/archives/C03G9H97MM2/p1728492002477169

This PR excludes the providers that aren't ready yet for building docs.

)

* Migrate the public endpoint Get DAG to FastAPI

* Use proper name for test function

* get dag_run init

* add serializer

* Merge branch 'main' of https:/apache/airflow into kalyan/AIP-84/get_dag_run

* add types

* add test

* working tests

* add note to DagRunResponse

* add note

* add test to test non Null note

* Update airflow/api_fastapi/views/public/dag_run.py

Co-authored-by: Pierre Jeambrun <[email protected]>

* Update airflow/api_fastapi/views/public/dag_run.py

Co-authored-by: Pierre Jeambrun <[email protected]>

* Merge branch 'main' of https:/apache/airflow into kalyan/AIP-84/get_dag_run

* add 404 test

---------

Co-authored-by: Pierre Jeambrun <[email protected]>

Co-authored-by: Majoros Donat (XC-DX/EET2-Bp) <[email protected]>

This actually does a little bit more.  It changes the backfill create endpoint to take a json payload instead of just query params.  This is just easier because we can use the backfill schema as the schema of the request body.

One thing that is maybe weird is I add a decorator to translate the request body to the kwargs in the endpoint function.  The main motivator here was for compatibility with the requires_access_dag decorator, which doesn't check request body.

apache#42916)

This method uses Backfill internally. Before we can remove BackfillJobRunner, we need to remove DAG.run. But before we can remove DAG.run, we need to update some old tests that use it. So this is the first step towards removing BackflilJobRunner.

There were some very old tests that came from airflow github issue 1225. These appeared to test the scheduler but really they tested the backfill job runner. Just to be cautious, I kept most of them rather than remove (which probably would have been fine since they essentially tested code that we'll be removing). As appropriate I either changed them to run on dag.test or scheduler. The ones dealing with ignore first depends on past will have to be added back when that functionality is implemented in new backfill.

-----
Co-authored-by: Kaxil Naik <[email protected]>

A couple of minor typos due to newlines

* Add possibility to override the conn type for Druid

Minor fix, which allows to use the schema which are specified in
theschema rather than `http` as default. In the same time it doesn't
changethe logic as any conn_type can be selected. Intuitevely it's
expectedthat anything specified in `schema` field will actually take
precedencein the building the desired url.

* Add druid endpoint connection from another PR

* Fix missing scheme in test

* Set schema to None where it's unused

Even though we don't need it directly set, by default the mock will set
it to an internal object, thus we need to override it to None.

---------

Co-authored-by: Oleg Auckenthaler <[email protected]>

…2925)

* Fix dag display name search

* Fix CI

…pache#42921)

* check _is_canary_run condition in is_legacy_ui_api_labeled method

* include pr check in is_legacy_ui_api_labeled

I'm not 100% sure why we added this check in the first place, but it doesn't
seem to be needed anymore (I've tested things locally with this removed and it
all seems to behave itself)

* Render errors when getting a list of dags

* Restore axios, prettierignore pnpm-store

* Add pnpm-store to prettier ignore

This will make sure we don't receive any information about the Webserver URL sending the info like the number of plugins and such.

…roject (apache#42505) (apache#42624)

This is only a partial split so far. It moves all the code and tests, but
leaves the creation of `core/` to a separate PR as this is already large
enough.

In addition to the straight file rename the other changes I had to make here
are:

- Some mypy/typing fixes.

  Mypy can be fragile about what it picks up when, so maybe some of those
  changes were caused by that. But the typing changes aren't large.

- Improve typing in common.sql type stub

  Again, likely a mypy file oddity, but the types should be safe

- Removed the `check-providers-init-file-missing` check

  This isn't needed now that airflow/providers shouldn't exist at all in the
  main tree.

- Create a "dev.tests_common" package that contains helper files and common
  pytest fixtures

  Since the provider tests are no longer under tests/ they don't automatically
  share the fixtures from the parent `tests/conftest.py` so they needed
  extracted.

  Ditto for `tests.test_utils` -- they can't be easily imported in provider
  tests anymore, so they are moved to a more explicit shared location.

In future we should switch how the CI image is built to make better use of UV
caching than our own approach as that would remvoe a lot of custom code.

Co-authored-by: Ash Berlin-Taylor <[email protected]>
Co-authored-by: Ryan Hatter <[email protected]>

Follow up after apache#42766 and apache#42936.

* We do not have to check for minimum Python version for Python 3.9
  any more (as we do not support 3.8 any more)
* While Python 3.12 is not yet fully supported by Apache Beam, we
  should still not allow it for releasing providers, but all other
  commands should support Python 3.12
* When you had pre-commit installed with Python 3.8 before, various
  errors might appear when running pre-commit. Troubleshooting
  was added quoting the errors and explaining what to do.

grammar update for clarity 😄 👍

I create a new command group "backfill" for management of backfills. The first action is "create" which creates a backfill. Some others may follow such as pause / cancel.

These links were pointing to the wrong location. This PR fixes it

…pache#42954)

* restrict looker version 24.18.0

* update microsoft provider deps with microsoft-kiota-http

* update gh issue ids to provider yaml files

* ignore type in applfowhook

* ignore type in applfowhook

* ignore type in applfowhook

apache#42960)

* Change code-block position

closes apache#42856

This PR adds a skeleton project for [AIP-72 Task SDK](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-72+Task+Execution+Interface+aka+Task+SDK)

It creates this new project, integrates them to breeze & runs it on CI.

The code & tests are just placeholders at this point, we will start adding things needs for DAG Authors in separate PRs.

Test was failing: https:/apache/airflow/actions/runs/11306055404/job/31446236535

```
pre-commit hook(s) made changes.
If you are seeing this message in CI, reproduce locally with: `pre-commit run --all-files`.
To run `pre-commit` as part of git workflow, use `pre-commit install`.
All changes made by hooks:
diff --git a/pyproject.toml b/pyproject.toml
index 844e1b6..5e9f587 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -30,7 +30,7 @@ requires = [
     "pluggy==1.5.0",
     "smmap==5.0.1",
     "tomli==2.0.2; python_version < '3.11'",
-    "trove-classifiers==2024.10.11",
+    "trove-classifiers==2024.10.12",
 ]
 build-backend = "hatchling.build"
```

A nit PR while I was looking at our doc builds

…d from an extended operator (apache#42849)

* refactor: Don't raise a warning when execute is called from an extended operator, as this should always be allowed.

* refactored: Fixed import of test_utils in test_dag_run

---------

Co-authored-by: David Blain <[email protected]>

There was a problem with CI image builds when cache has been
disabled - they still used "pre-cached" main version (unlike
PROD builds). This PR synchronizes the behaviour between CI
and PROD builds.

For removing dependencies, you need to set both:

* `disable docker cache` label
* incerase DEPENDENCIES_EPOCH_NUMBER in `Dockerfile.ci`

Comments and documentation in both places has been updated to
reflect it.

Since documentation for labels has been updated,
Part of this PR is to improve the description of possible
labels that could be used during the build.
The description grew from a small number of labels to a
"wall of text" that was difficult to read. This PR reformats it in the
form of table that makes it far easier to see actions that the
maintainer can do and what labels should be set for each of them.

…tem (apache#42964)

* delete dag_run init

* delete dag_run init

* add delete dag run endpoint

* add tests for delete dag run

* Make datStats dag_id parameter optional with pagination

* move pagination out of sql query

* tidy

---------

Co-authored-by: Michael Smith-Chandler <[email protected]>

…ords (apache#42860)

* Change directory used by simple auth manager to store generated passwords

* Update airflow/auth/managers/simple/simple_auth_manager.py

Co-authored-by: Jens Scheffler <[email protected]>

---------

Co-authored-by: Jens Scheffler <[email protected]>

* quarantined flaky mssql integration tests

* disable mssql from ci integration checks