Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Added event_handler parameter in MSGraphAsyncOperator #42539

Merged
merged 50 commits into from
Oct 15, 2024

Conversation

dabla
Copy link
Contributor

@dabla dabla commented Sep 27, 2024

Added an event_handler parameter in MSGraphAsyncOperator which allows you to override the default implementation when needed. This can be handy when you want to handle certain failed event types returned by the triggerer. Also added a unit test for this new feature and updated docstring.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@dabla dabla marked this pull request as draft September 27, 2024 09:25
@dabla dabla marked this pull request as ready for review September 29, 2024 11:19
@dabla
Copy link
Contributor Author

dabla commented Oct 7, 2024

Dunno why following tests keeps failing, it succeeds locally:

FAILED tests/providers/microsoft/azure/operators/test_data_factory.py::TestAzureDataFactoryRunPipelineOperator::test_execute_wait_for_termination[Canceling-timeout] - AssertionError: assert 2 == 4
 +  where 2 = <MagicMock name='get_pipeline_run' id='139745815447776'>.call_count
=========== 1 failed, 6661 passed, 60 skipped in 1235.68s (0:20:35) ============
/usr/local/lib/python3.8/site-packages/pytest_asyncio/plugin.py:208: PytestDeprecationWarning: The configuration option "asyncio_default_fixture_loop_scope" is unset.
The event loop scope for asynchronous fixtures will default to the fixture caching scope. Future versions of pytest-asyncio will default the loop scope for asynchronous fixtures to function scope. Set the default fixture loop scope explicitly in order to avoid unexpected behavior in the future. Valid fixture loop scopes are: "function", "class", "module", "package", "session"

@potiuk
Copy link
Member

potiuk commented Oct 7, 2024

Did you try reproducing it with upgraded boto:

If you unfold the failed output you will see this:

  Upgrading boto3, botocore to latest version to run Amazon tests with them
  
  Uninstalled 4 packages in 395ms
   - aiobotocore==2.15.1
   - opensearch-py==2.7.1
   - s3fs==2024.9.0
   - yandexcloud==0.320.0
  + uv pip install --python /usr/local/bin/python --upgrade boto3 botocore 'oss2>=2.14.0' 'cryptography<43.0.0' 'requests!=2.32.*,<3.0.0,>=2.24.0'
  Resolved 19 packages in 2.87s
  Prepared 3 packages in 438ms
  Uninstalled 3 packages in 882ms
  Installed 3 packages in 139ms
   - boto3==1.35.23
   + boto3==1.35.34
   - botocore==1.35.23
   + botocore==1.35.34
   - requests==2.32.3
   + requests==2.31.0

Which can be locally reproduced with:

breeze shell --upgrade-botot

And then rerunning the tests.

@potiuk
Copy link
Member

potiuk commented Oct 7, 2024

(It might also be transitive / accidental side effect)

@dabla
Copy link
Contributor Author

dabla commented Oct 8, 2024

(It might also be transitive / accidental side effect)

Apparently it was, it failed for days even after updates now it passes :)

@potiuk
Copy link
Member

potiuk commented Oct 14, 2024

It looks that there were some changes in looker/kiota packages recently. Can you please rebase again and resolve conflicts and see where it is and ping me if you still see errors ?

# Conflicts:
#	generated/provider_dependencies.json
#	providers/src/airflow/providers/microsoft/azure/provider.yaml
@dabla
Copy link
Contributor Author

dabla commented Oct 14, 2024

It looks that there were some changes in looker/kiota packages recently. Can you please rebase again and resolve conflicts and see where it is and ping me if you still see errors ?

Resolved conflicts, will see how the build behaves.

@dabla
Copy link
Contributor Author

dabla commented Oct 14, 2024

It looks that there were some changes in looker/kiota packages recently. Can you please rebase again and resolve conflicts and see where it is and ping me if you still see errors ?

Everything is fine now after update and conflict resolution

@potiuk potiuk merged commit 2b101e2 into apache:main Oct 15, 2024
103 checks passed
@potiuk
Copy link
Member

potiuk commented Oct 15, 2024

Woooohooo!

@dabla dabla deleted the feature/msgraph-event-handler branch October 15, 2024 07:35
R7L208 pushed a commit to R7L208/airflow that referenced this pull request Oct 17, 2024
…e#42539)

* refactor: Added parameter in MSGraphAsyncOperator to allow overriding default event_handler

* docs: Added docstring for event_handler parameter in MSGraphAsyncOperator

* refactor: Fixed TestMSGraphAsyncOperator

* refactor: Check if event is not None

* refactor: Register the TextParseNodeFactory and JsonParseNodeFactory so error messages get handled correctly in RequestAdapter

* refactor: Reorganized import for TestMSGraphAsyncOperator

* refactor: Added missing kiota-serialization packages in azure provider

* refactor: Updated provider dependencies

* refactor: Reorganized import of TestKiotaRequestAdapterHook

* refactor: Downgraded version of json kiota serialization

* refactor: Updated provider dependencies

* refactor: Put import of Context in TYPE_CHECKING block

* refactor: Fixed lookup of tenant-id

* refactor: Fixed kiota serialization dependencies to 1.0.0 to avoid pendulum dependency issues for backward compatibility

* refactor: Updated provider dependencies

* refactored: Fixed import of test_utils in test_dag_run

---------

Co-authored-by: David Blain <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants