Add opentelemetry callback plugin #3091

v1v · 2021-07-27T08:14:51Z

SUMMARY

Send distributed traces for the ansible runs with OpenTelemetry

ISSUE TYPE

New Module Pull Request

COMPONENT NAME

plugins/callback/opentelemetry.py

ADDITIONAL INFORMATION

You can configure this plugin and then run your playbook with the below environment variables. Those environment variables are the ones defined by the opentelemetry project (see https://opentelemetry-python.readthedocs.io/en/latest/exporter/otlp/otlp.html)

In order to configure this plugins is requried to install the below python libs:

pip3 install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp

Then you can start sending traces to your OTEL endpoint:

$ OTEL_SERVICE_NAME='ansible-otel' \
   OTEL_EXPORTER_OTLP_ENDPOINT='<your_otlp_endpoint>' \
   OTEL_EXPORTER_OTLP_HEADERS='authorization=Bearer <your_token>' \
   ansible-playbook <your_playbook>

Or even you can see the distributed traces in the console:

$ OPENTELEMETRY_CONSOLE_OUTPUT=true \
   ansible-playbook <your_playbook>

This is vendor agnostic, as long as the OpenTelemetry is supported those traces are sent, for instance, when running the molecule test the converge stage produced the below traces for one of our roles.

Molecule was configured with something like:

provisioner:
  name: ansible
  env:
    JUNIT_OUTPUT_DIR: $MOLECULE_SCENARIO_DIRECTORY/test-results
  config_options:
    defaults:
      callback_plugins: $MOLECULE_SCENARIO_DIRECTORY/../../../../callback_plugins
      callback_whitelist: opentelemetry

Questions

Can I add some ITs to help?

Tasks

Add tests
Verify changes locally.
Verify changes in a staging environment.

Follow ups

The below follow ups will be tackle in future PRs after merging this

Add context propagation
Handle task failures as errors

felixfontein

Thanks for your contribution! I've added some first comments.

Please also check the failing sanity tests.

Also please note that all new plugins and modules must come with tests.

plugins/callback/opentelemetry.py

tests/unit/plugins/callback/test_opentelemetry.py

plugins/callback/opentelemetry.py

v1v · 2021-09-03T07:43:37Z

/rebuild_failed

codecov · 2021-09-03T08:33:31Z

Codecov Report

Merging #3091 (3502f3b) into main (118c040) will decrease coverage by 0.17%.
The diff coverage is n/a.

❗ Current head 3502f3b differs from pull request most recent head deb84a6. Consider uploading reports for the commit deb84a6 to get more accurate results

@@            Coverage Diff             @@
##             main    #3091      +/-   ##
==========================================
- Coverage   43.56%   43.38%   -0.18%     
==========================================
  Files         928      923       -5     
  Lines       89375    89069     -306     
  Branches    16284    16265      -19     
==========================================
- Hits        38933    38646     -287     
+ Misses      48737    48720      -17     
+ Partials     1705     1703       -2

Impacted Files	Coverage Δ
plugins/modules/system/pids.py	`78.18% <0.00%> (-8.43%)`	⬇️
plugins/modules/files/iso_create.py	`75.23% <0.00%> (-3.81%)`	⬇️
plugins/modules/storage/zfs/zpool_facts.py	`28.07% <0.00%> (-3.31%)`	⬇️
plugins/modules/net_tools/nmcli.py	`79.21% <0.00%> (-1.47%)`	⬇️
...modules/source_control/gitlab/test_gitlab_group.py	`78.57% <0.00%> (-1.16%)`	⬇️
tests/unit/plugins/modules/net_tools/test_nmcli.py	`99.89% <0.00%> (-0.01%)`	⬇️
plugins/inventory/linode.py	`47.11% <0.00%> (ø)`
...it/plugins/modules/source_control/gitlab/gitlab.py	`83.98% <0.00%> (ø)`
...gins/modules/database/misc/test_redis_data_info.py
plugins/module_utils/redis.py
... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bd63da6...deb84a6. Read the comment docs.

basepi

Looks awesome! 👍

plugins/callback/opentelemetry.py

ansibullbot · 2021-09-06T22:05:13Z

@v1v this PR contains the following merge commits:

8acbbe9

Please rebase your branch to remove these commits.

click here for bot help

This reverts commit 3a54c827c5472553a6baf5598bc76a0f63f020c1.

This reverts commit cab9d8648899c28c0345745690c4ec7a41f7e680.

Co-authored-by: flowerysong <[email protected]>

patchback · 2021-09-14T19:05:06Z

Backport to stable-3: 💚 backport PR created

✅ Backport PR branch: patchback/backports/stable-3/517570a64fcbbfdad5704f88c63eabb4cf74a84d/pr-3091

Backported as #3373

🤖 @patchback
I'm built with octomachinery and
my source is open — https:/sanitizers/patchback-github-app.

* Add opentelemetry callback plugin * Apply suggestions from code review Co-authored-by: Felix Fontein <[email protected]> * Formatting (text), booleans and renamed env variables * This should be done in a future release * Remove insecure in favour of the OTEL env variable. Add descriptions * Use OpenTelemetrySource * Move generate_distributed_traces * Move update_span_data and set_span_attribute * Move finish_task * Move start_task * Refactor to support UTs * Add first UT * Fix codestyle * opentelemetry callback entry in the botmeta * Fix linting * Fix signature * Mock methods * Use MagicMock * Mock the methods * UT for transform_to_boolean_or_default * Fix linting * Set test data * Mock _time_ns * Exclude tests for python <= 3.6 * Remove obsoleted setup task type configuration * Remove unused docs * Apply suggestions from code review Co-authored-by: Felix Fontein <[email protected]> * Fix docs * unrequired logic that was originally took from https:/ansible/ansible/blob/devel/lib/ansible/plugins/callback/junit.py\#L226 * Use raise_from for the required dependencies * Fix linting * Add requirements for the UTs * add missing dependency for the opentelemetry plugin in the UTs * Add ANSIBLE_ prefix for the ansible specific options * Add more context in the docs and remove duplicated docs * As suggested in the code review * Verify if the OTEL env variables for the endpoint were set * Fix docs typo * Fix linting * Revert "Fix linting" This reverts commit 3a54c827c5472553a6baf5598bc76a0f63f020c1. * Revert "Verify if the OTEL env variables for the endpoint were set" This reverts commit cab9d8648899c28c0345745690c4ec7a41f7e680. * Remove console_output as suggested * Apply suggestions from code review Co-authored-by: flowerysong <[email protected]> * Delegate the definition of OTEL_EXPORTER_OTLP_INSECURE to the user * Move definitions above, close to the class that uses them Co-authored-by: Felix Fontein <[email protected]> Co-authored-by: flowerysong <[email protected]> (cherry picked from commit 517570a)

felixfontein · 2021-09-14T19:05:28Z

@v1v thanks for contributing this plugin!
@basepi @flowerysong thanks for reviewing this!

* Add opentelemetry callback plugin * Apply suggestions from code review Co-authored-by: Felix Fontein <[email protected]> * Formatting (text), booleans and renamed env variables * This should be done in a future release * Remove insecure in favour of the OTEL env variable. Add descriptions * Use OpenTelemetrySource * Move generate_distributed_traces * Move update_span_data and set_span_attribute * Move finish_task * Move start_task * Refactor to support UTs * Add first UT * Fix codestyle * opentelemetry callback entry in the botmeta * Fix linting * Fix signature * Mock methods * Use MagicMock * Mock the methods * UT for transform_to_boolean_or_default * Fix linting * Set test data * Mock _time_ns * Exclude tests for python <= 3.6 * Remove obsoleted setup task type configuration * Remove unused docs * Apply suggestions from code review Co-authored-by: Felix Fontein <[email protected]> * Fix docs * unrequired logic that was originally took from https:/ansible/ansible/blob/devel/lib/ansible/plugins/callback/junit.py\#L226 * Use raise_from for the required dependencies * Fix linting * Add requirements for the UTs * add missing dependency for the opentelemetry plugin in the UTs * Add ANSIBLE_ prefix for the ansible specific options * Add more context in the docs and remove duplicated docs * As suggested in the code review * Verify if the OTEL env variables for the endpoint were set * Fix docs typo * Fix linting * Revert "Fix linting" This reverts commit 3a54c827c5472553a6baf5598bc76a0f63f020c1. * Revert "Verify if the OTEL env variables for the endpoint were set" This reverts commit cab9d8648899c28c0345745690c4ec7a41f7e680. * Remove console_output as suggested * Apply suggestions from code review Co-authored-by: flowerysong <[email protected]> * Delegate the definition of OTEL_EXPORTER_OTLP_INSECURE to the user * Move definitions above, close to the class that uses them Co-authored-by: Felix Fontein <[email protected]> Co-authored-by: flowerysong <[email protected]> (cherry picked from commit 517570a) Co-authored-by: Victor Martinez <[email protected]>

felixfontein reviewed Jul 27, 2021

View reviewed changes

This was referenced Aug 18, 2021

[Enhancement] Create binaries of the otel-cli to ease adoption equinix-labs/otel-cli#37

Closed

Environment Variables as Carrier for Inter-Process Propagation to transport context open-telemetry/opentelemetry-specification#740

Open

ansibullbot added the has_issue label Aug 30, 2021

felixfontein added backport-3 check-before-release PR will be looked at again shortly before release and merged if possible. labels Aug 31, 2021

v1v commented Sep 1, 2021

View reviewed changes

tests/unit/plugins/callback/test_opentelemetry.py Show resolved Hide resolved

v1v commented Sep 2, 2021

View reviewed changes

plugins/callback/opentelemetry.py Outdated Show resolved Hide resolved

v1v marked this pull request as ready for review September 2, 2021 16:24

basepi approved these changes Sep 3, 2021

View reviewed changes

ansibullbot added tests tests unit tests/unit labels Sep 4, 2021

felixfontein reviewed Sep 6, 2021

View reviewed changes

plugins/callback/opentelemetry.py Outdated Show resolved Hide resolved

plugins/callback/opentelemetry.py Show resolved Hide resolved

plugins/callback/opentelemetry.py Outdated Show resolved Hide resolved

plugins/callback/opentelemetry.py Outdated Show resolved Hide resolved

ansibullbot added needs_rebase https://docs.ansible.com/ansible/devel/dev_guide/developing_rebasing.html needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR and removed community_review labels Sep 6, 2021

ansibullbot added merge_commit This PR contains at least one merge commit. Please resolve! and removed needs_revision This PR fails CI tests or a maintainer has requested a review/revision of the PR labels Sep 6, 2021

v1v and others added 15 commits September 14, 2021 09:01

Fix linting

bc7efbb

Add requirements for the UTs

7e84d83

add missing dependency for the opentelemetry plugin in the UTs

301679a

Add ANSIBLE_ prefix for the ansible specific options

5761800

Add more context in the docs and remove duplicated docs

1069b46

As suggested in the code review

305dbb6

Verify if the OTEL env variables for the endpoint were set

06f06e2

Fix docs typo

fe83d7a

Fix linting

3c77baf

Revert "Fix linting"

09e2a6c

This reverts commit 3a54c827c5472553a6baf5598bc76a0f63f020c1.

Revert "Verify if the OTEL env variables for the endpoint were set"

62bfc83

This reverts commit cab9d8648899c28c0345745690c4ec7a41f7e680.

Remove console_output as suggested

075f3e1

Apply suggestions from code review

af1bd70

Co-authored-by: flowerysong <[email protected]>

Delegate the definition of OTEL_EXPORTER_OTLP_INSECURE to the user

86ed157

Move definitions above, close to the class that uses them

deb84a6

v1v force-pushed the feature/add-opentelemetry-callback-plugin branch from 4130319 to deb84a6 Compare September 14, 2021 08:02

felixfontein removed the check-before-release PR will be looked at again shortly before release and merged if possible. label Sep 14, 2021

felixfontein merged commit 517570a into ansible-collections:main Sep 14, 2021

patchback bot mentioned this pull request Sep 14, 2021

[PR #3091/517570a6 backport][stable-3] Add opentelemetry callback plugin #3373

Merged

3 tasks

cyrille-leclerc mentioned this pull request May 22, 2024

Integrate Jenkins OpenTelemetry with Gradle OpenTelemetry jenkinsci/opentelemetry-plugin#850

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add opentelemetry callback plugin #3091

Add opentelemetry callback plugin #3091

v1v commented Jul 27, 2021 •

edited

Loading

felixfontein left a comment

v1v commented Sep 3, 2021

codecov bot commented Sep 3, 2021 •

edited

Loading

basepi left a comment

ansibullbot commented Sep 6, 2021

patchback bot commented Sep 14, 2021 •

edited

Loading

felixfontein commented Sep 14, 2021

Add opentelemetry callback plugin #3091

Add opentelemetry callback plugin #3091

Conversation

v1v commented Jul 27, 2021 • edited Loading

SUMMARY

ISSUE TYPE

COMPONENT NAME

ADDITIONAL INFORMATION

Questions

Tasks

Follow ups

felixfontein left a comment

Choose a reason for hiding this comment

v1v commented Sep 3, 2021

codecov bot commented Sep 3, 2021 • edited Loading

Codecov Report

basepi left a comment

Choose a reason for hiding this comment

ansibullbot commented Sep 6, 2021

patchback bot commented Sep 14, 2021 • edited Loading

Backport to stable-3: 💚 backport PR created

felixfontein commented Sep 14, 2021

v1v commented Jul 27, 2021 •

edited

Loading

codecov bot commented Sep 3, 2021 •

edited

Loading

patchback bot commented Sep 14, 2021 •

edited

Loading