Datadog and opentelemetry-cpp

[dgoffredo]

Datadog is interested in integrating with opentelemetry-cpp. Previously we integrated with OpenTracing via dd-opentracing-cpp. We now have a dedicated core tracing library called dd-trace-cpp that does not implement any API.

Currently, Datadog customers can instrument their C++ code using opentelemetry-cpp and then configure the library to send traces to the Datadog Agent, which speaks OTLP.

However, this precludes several Datadog-specific tracing features. An example of one such feature is the ways in which trace sampling can be configured.

Other programming language integrations, such as Datadog's Go tracer, implement the "tracer" and "tracer provider" interfaces in a separate library that customers can then depend on to provide a Datadog-specific tracer for their otherwise OpenTelemetry-generic code.

Integrating with opentelemetry-cpp is important not only for customers using OpenTelemetry programmatically, but also for users of HTTP proxies whose tracing will be provided by OpenTelemetry, such as the NGINX Ingress Controller for Kubernetes. In fact, tracing support within HTTP proxies is my primary concern.

Which approach do you recommend for adapting the capabilities of dd-trace-cpp into OpenTelemetry?

I've done a cursory exploration of several options, which I'm happy to discuss here in more detail.

[lalitb]

I've done a cursory exploration of several options, which I'm happy to discuss here in more detail.

Would be interested to hear the options you have explored, either as comment here, or in general if you would like to join the community meeting to discuss further :)
As of now, you are right, there is no direct way to plugin a external Tracer object in otel-cpp SDK.

[dgoffredo]

@lalitb The two best candidates so far are:

1. Implement TracerProvider and the related interfaces in a third-party library.
   • Users of OpenTelemetry, such as otel_ngx_module, include both
     opentelemetry-cpp and the third-party library in their build.
   • main(), or the relevant telemetry initialization function,
     optionally uses Datadog's TracerProvider from the third-party
     library.
   • All other user code is expressed in terms of opentelemetry-cpp's public
     API.
2. Implement TracerProvider and the related interfaces within opentelemetry-cpp.
   • This means that dd-trace-cpp would be added to opentelemetry-cpp's
     dependencies.
   • This is similar to what is done for Event Tracing for Windows.
   • This approach is what is done for Envoy.
   • Users need depend on opentelemetry-cpp only, and select the desired
     TracerProvider at runtime per (1).

All of the other options that I looked into have a similar user experience
but are technically inferior, e.g. defining a third-party SpanProcessor
that creates Datadog-specific spans in the background. Not ideal, and leaves
out control of propagation.

Another, completely different, approach would be to extend the OpenTelemetry
API so that currently-Datadog-specific features could be expressed within the
API -- the default implementations in the SDK would then also be extended to
be capable of everything done in dd-trace-cpp. I think that this would be
an unwise coupling between Datadog specifics and the larger telemetry
ecosystem.

What do you think of these approaches? How should we get Datadog aboard the
opentelemetry-cpp train?

If either (1) or (2) is viable, then I have a preference for (2). However, (1)
is what is done in Go.

[lalitb]

@dgoffredo thanks for providing the alternatives. These was also shortly discussed in community meeting. I would suggest
option (1) here. The opentelemetry-cpp repo is supposed to have - API, SDK, and exporters as specified in otel specs. There are some exceptions - ETW and ElasticSearch exporters, but we plan to move them to otel-cpp-contrib repo as tracked in #1423, and #1424.

[lalitb]

@dgoffredo should this be good to close, or if you have some other asks ?

[dgoffredo]

Yes, this is good to close. I'll mention it in a future issue in opentelemetry-cpp-contrib, but we've cleared up the topic here, I think: Datadog can create its own implementation of the opentelemetry-cpp API using the opentelemetry-cpp SDK, and host the implementation in a separate repository.