-
Notifications
You must be signed in to change notification settings - Fork 14.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create simple "Task API" server based on FastAPI #43009
Comments
I think it really depends on how we want to treat providers - this is the big difference of dependencies. Technically - if we solve base links and connection issue, we could have the same dependencies for both - none of the task api calls nor web API should need providers at all in this case. On the other hand, if we don't solve it, then both will have providers if they are run together - but task API technically could be separate and have no provider dependencies. I think we should aim on having the same dependencies in both for simplicity, and we should see if we can remove providers from both - but this can also be done later - either before 3.0 or after - this only changes security property of the deployment, so it won't be a breaking change. |
Yea, we absolutely need to solve Operator links & Connection deps. Jens & Vikram have a proposal. One of the last dev calls (Sep 19) we discussed about routing it through an API (and stored it in DB): |
closes apache#43009 The Task Execution API is created as a separate FastAPI APP so that it allows us to hook CLI and selectively run the "Execution API" with the rest of Airflow API (UI/public). This is so that users can scale the APIs separately if needed as mentioned in apache#43009
closes apache#43009 The Task Execution API is created as a separate FastAPI APP so that it allows us to hook CLI and selectively run the "Execution API" with the rest of Airflow API (UI/public). This is so that users can scale the APIs separately if needed as mentioned in apache#43009
Need to work with Pierre on this to work out the right code and URI layout for this.
I think there have been some people who want to run them separately (it was certainly asked about by someone in my talk), so I think they should be structured as two separate FastAPI/ASGI apps that can optionally be run in a single binary (i.e. ASGI "mounting" etc.). At this stage I'm thinking that we should not separate out the dependencies/make them stand-alone projects (that can happen later if there proves to be enough desire)
The text was updated successfully, but these errors were encountered: