Support Akka actor metrics in otel extension

[lgajowy]

Similar to #291 and #296.

[lgajowy]

We can't add new fields to classes - the otel agent forbids it. To be investigated if we can use: VirtualField from opentelemetry along with otel context etc. Below are otel agent's instrumentations that use VirtualFIelds:

https:/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/akka/akka-actor-2.5/javaagent/src/main/java/io/opentelemetry/javaagent/instrumentation/akkaactor

[lgajowy]

Virtual field class: https:/open-telemetry/opentelemetry-java-instrumentation/blob/main/instrumentation-api/src/main/java/io/opentelemetry/instrumentation/api/field/VirtualField.java

[lgajowy]

In opentelemetry, there is a whole Instrumenter api that allows attaching metric instruments via its addRequestMetrics method. Such instruments are just otel instruments that we could use to report the akka-actor metrics (I think).

Instrumenter: link

RequestListener link (for metric attaching)

[lgajowy]

The instrumenter/opentelemetry context api sounds like a good idea for the future, but it requires lots of code rewriting. It's planned either way but requires more research.

For now it makes sense to just use a WeakHashMap for the Metrics obiects added by actorCell instrumentation and get rid of creating variables this way. As this seems to be the only blocker when it comes to using akkaActorAgent code in otel extension, I will proceed this way instead.

[lgajowy]

For now it makes sense to just use a WeakHashMap for the Metrics obiects added by actorCell instrumentation and get rid of creating variables this way. As this seems to be the only blocker when it comes to using akkaActorAgent code in otel extension, I will proceed this way instead.

Actually using some kind of WeakHashMap (that would need be concurrent btw) is the thing that a VirtualField does underneath, so going the way described above would be reinventing the VirtualFields from opentelemetry.

We can't add new fields to classes - the otel agent forbids it.

I will go and investigate the use of the VirtualFields for replacing the places where we needed to add fields to classes. It seems it's as easy as:

• using VirtualField.find(ActorContext.class, ActorCellMetrics.class).set(key, value); wherever we need to set the cell metrics
• using VirtualField.find(ActorContext.class, ActorCellMetrics.class).get(key, value); when we want to access that field.

The VirtualFIelds use a "Weak Cache" underneath. That means that they are:

• backed by weak references - this should prevent memory leaks
• thread-safe (and it was optimized recently)

To be checked:

• do they work without the otel agent (if not, we may need to decide to drop the Mesmer agent earlier, or have 2 different versions of akka agent instrumentation. We can also drop the mesmer agent support since it's planned to happen eventually).
• how can we initialize metrics properly? We previously used an byte-byddy-injected "setupMetrics" method in ActorCell. Since we don't want to do that, when should we initialize particular metrics? Options I see:
• do it lazily on first use: that afaik would mean that we won't see a metric in a dashboard until first record is read but maybe that's ok
• initialize them along with instrumentations that need them (is it always possible to access ActorCell/ActorContext so that we can get the virtual field when needed?)

FYI @worekleszczy