[PR #3091/517570a6 backport][stable-3] Add opentelemetry callback plugin

[patchback]

This is a backport of PR #3091 as merged into main (517570a).

SUMMARY

Send distributed traces for the ansible runs with OpenTelemetry

ISSUE TYPE

• New Module Pull Request

COMPONENT NAME

plugins/callback/opentelemetry.py

ADDITIONAL INFORMATION

You can configure this plugin and then run your playbook with the below environment variables. Those environment variables are the ones defined by the opentelemetry project (see https://opentelemetry-python.readthedocs.io/en/latest/exporter/otlp/otlp.html)

In order to configure this plugins is requried to install the below python libs:

pip3 install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp

Then you can start sending traces to your OTEL endpoint:

$ OTEL_SERVICE_NAME='ansible-otel' \
   OTEL_EXPORTER_OTLP_ENDPOINT='<your_otlp_endpoint>' \
   OTEL_EXPORTER_OTLP_HEADERS='authorization=Bearer <your_token>' \
   ansible-playbook <your_playbook>

Or even you can see the distributed traces in the console:

$ OPENTELEMETRY_CONSOLE_OUTPUT=true \
   ansible-playbook <your_playbook>

This is vendor agnostic, as long as the OpenTelemetry is supported those traces are sent, for instance, when running the molecule test the converge stage produced the below traces for one of our roles.

Molecule was configured with something like:

provisioner:
  name: ansible
  env:
    JUNIT_OUTPUT_DIR: $MOLECULE_SCENARIO_DIRECTORY/test-results
  config_options:
    defaults:
      callback_plugins: $MOLECULE_SCENARIO_DIRECTORY/../../../../callback_plugins
      callback_whitelist: opentelemetry

Questions

• Can I add some ITs to help?

Tasks

• Add tests
• Verify changes locally.
• Verify changes in a staging environment.

Follow ups

The below follow ups will be tackle in future PRs after merging this

1. Add context propagation
2. Handle task failures as errors