Missing traces/spans for requests

[cbos]

We have a third party docker containers were we have injected the opentelemetry java agent.
That works great. Metrics, logging are extracted.

But not for all requests (actual the requests I want to know more about), are not traced.

This is information from the metrics (prometheus endpoint):
Irrelevant information is stripped

http_server_duration_count{http_flavor="1.1",http_host="internal_hostname:8081",http_method="POST",http_scheme="http",http_status_code="200",service_name="myservice"} 1366 1650014515741
http_server_duration_count{http_flavor="1.1",http_host="x_internal_ip.x.x.x:8081",http_method="POST",http_route="/api/v1/aangiftes/berekening",http_scheme="http",http_status_code="500",service_name="myservice"} 1 1650014515741
http_server_duration_count{http_flavor="1.1",http_host="localhost:8081",http_method="OPTIONS",http_route="/*",http_scheme="http",http_status_code="202",service_name="myservice"} 1187 1650014515741
http_server_duration_count{http_flavor="1.1",http_host="internal_hostname:8081",http_method="POST",http_route="/api/v1/aangiftes/berekening",http_scheme="http",http_status_code="500",service_name="myservice"} 1 1650014515741
otlp_exporter_exported{service_name="myservice",success="true",type="span"} 1191 1650014515741
otlp_exporter_seen{service_name="myservice",type="span"} 1191 1650014515741
processedSpans{dropped="false",service_name="myservice",spanProcessorType="BatchSpanProcessor"} 1191 1650014515741

I see a number of requests without an http_route.
These requests do not get traced.
All the other requests are send either curl or by the monitoring (the OPTIONS requests). These requests do get traced.

The third party Java service is using Spring boot with Jersey JaxRS.

The invocation of these services is done by .Net code in Azure. HttpClient postAsync with a payload.
These are also send to /api/v1/aangiftes/berekening, but apparently not recognized for that http_route.

My question is:
How can we get the metrics right, then it also gets the http_route and how do we get the traces for these requests as well.

[mateuszrzeszutek]

Hey @cbos ,
Can you create a small example app that reproduces your issue? Without a repro scenario it's really hard to debug/fix these things - especially so when it comes to the http.route, because different instrumentations are responsible for creating server spans and setting the route.

[cbos]

@mateuszrzeszutek
I will be working on that.
First step is done, I found a way to reproduce this issue at least with a client.

The 'problem' is with httpClient factory in .Net.
With just using HttpClient I got not reproduce the problem.

Now I created a simple ASP.NET setup with by using httpClient factory.
https://docs.microsoft.com/en-us/dotnet/architecture/microservices/implement-resilient-applications/use-httpclientfactory-to-implement-resilient-http-requests

When I do now a request, I see the same problem (comparable with our setup on production).

Next steps will follow.

[mateuszrzeszutek]

No, the client shouldn't really matter - what's most important is your server setup: what libraries/frameworks/servers you're using, how your app is structured, etc.

[cbos]

@mateuszrzeszutek
True that is what I hoped as well, but I was not able to reproduce it curl or browser ... :(
But apparently there is a difference.
I have at least that part 'reproduced', I will see if I get more logging from the setup to see more information about the requests and why that is 'different'.

[cbos]

@mateuszrzeszutek
I found the next piece in the puzzle, which I found with the debug logs of the request.

# this requests gives a HTTP 500 response (as expected)
# Trace is available in this case and in the Metrics there is a http_route
curl -X POST http://localhost:8081/api/v1/aangiftes/berekening

# By adding a header traceparent, there will be no trace and http_route is missing for the request
curl -v -H "traceparent: 00-9c39351ea30b774f80b9f5bc962ad627-d26441d873ab4843-00" -X POST http://localhost:8081/api/v1/aangiftes/berekening

As soon as the traceparent header is added the data is missing.

Is this information enough for this ticket? Or do you need a test application?

[mateuszrzeszutek]

Ah, that helps, thanks! I think I know where the problem is.

Looks like your request is not sampled the ...-00 part in the traceparent header represents a "not sampled" flag; by default, a span whose parent is not sampled won't be sampled as well, and won't be exported.
But this should not affect metrics - even if a span is not sampled, we should still set the http.route attribute on server metrics properly. I believe this is a bug

[cbos]

As this these requests come from an upstream services, the traceparent header is send.
Still we would like to get the spans from these services. Is there a way we can overrule this and still get the spans exported?

The services I am busy with are not called that often, but give sometimes some weird behavior, so I like to get as much as detailed information out of it.

[mateuszrzeszutek]

Yes, you can set the otel.traces.sampler to always_on (the default setting is parentbased_always_on) - this will make the sampler ignore the sampled flag that comes with the parent span context, and always sample the span.
Note that this will make incomplete traces appear in your tracing backend (parent span was not sampled and not exported, child span that references the parent span was sampled and exported).

[cbos]

Thanks for fixing this issue!

Any idea when a new release will become available?

[trask]

hi @cbos! current target for 1.13.0 release is Monday.

[cbos]

@trask @mateuszrzeszutek
Thanks, I just started using 1.13.0 and the issue is solved!