agent.tags parsing error for Elastic Cloud

[Aaronontheweb]

Describe the bug
While following the instructions for exporting spans from Jaeger Agents to Elastic Cloud elastic/apm#212, I've not been able to get the recommended agent.tags format for the given agent to parse correctly using CLI arguments or the JAEGER_TAGS environment variable for the client.

--reporter.grpc.tls.enabled=true
--reporter.grpc.host-port=$(ELASTIC_APM_URI)
--agent.tags="elastic-apm-auth=Bearer ${ELASTIC_APM_TOKEN}"

The agent starts up fine, but I get the following error whenever we attempt to transmit spans over gRPC:

{"level":"error","ts":1635462398.3329742,"caller":"grpc/reporter.go:74","msg":"Could not send spans over gRPC","error":"rpc error: code = PermissionDenied desc = unauthorized: anonymous access not permitted for agent

This is because the bearer token included in the span tags was not propagated in the correct format - even if I hard code the ELASTIC_APM_TOKEN value into the agent.tags value.

However, if I make the following call using the Jaeger C# client:

var tracer = new Tracer.Builder(typeof(Startup).Assembly.GetName().Name)
                    .WithReporter(new CompositeReporter(remoteReporter, logReporter))
                    .WithSampler(sampler)
                    .WithScopeManager(
                        new ActorScopeManager())
                    .WithTag("elastic-apm-auth", "Bearer <hard coded value>");

Then that works fine and my spans arrive in Elastic Cloud no problem.

To Reproduce
Create an agent using the following configuration (using Kubernetes in my case):

containers:
 - name: jaeger-agent
        image: jaegertracing/jaeger-agent:1.27
        ports:
        - containerPort: 5775
          protocol: UDP
        - containerPort: 5778
          protocol: TCP
        - containerPort: 6831
          protocol: UDP
        - containerPort: 6832
          protocol: UDP
       - name: ELASTIC_APM_URI
          valueFrom:
            secretKeyRef:
              name: elastic-secrets
              key: ELASTIC_APM_URI
        - name: ELASTIC_APM_TOKEN
          valueFrom:
            secretKeyRef:
              name: elastic-secrets
              key: ELASTIC_APM_TOKEN
        args:
          - --reporter.grpc.tls.enabled=true
          - --reporter.grpc.host-port=$(ELASTIC_APM_URI)
          - --agent.tags="elastic-apm-auth=Bearer ${ELASTIC_APM_TOKEN}"
        resources:
          limits:
            memory: 50M
            cpu: 100m
          requests:
            memory: 50M
            cpu: 100m

Have a client propagate spans to it while using the elastic-apm-auth=Bearer ${ELASTIC_APM_TOKEN} tag - and if connected to Elastic Cloud you'll receive an authorization error. I'm not sure why you will - passing in the tag explicitly via the programmatic client configuration works fine, but not if you do it via environment variables or via the agent.tags property.

Expected behavior
The content and the format of the elastic-apm-auth=Bearer ${ELASTIC_APM_TOKEN} tag should be preserved.

Edit: fixed my YAML - had the relevant part commented out while I was troubleshooting.

[yurishkuro]

--agent.tags is for enriching spans (which are payload), it has nothing to do with propagation of the bearer tokens (which exists at the protocol level)

[Aaronontheweb]

@yurishkuro I fully agree, but it's what the Elastic folks are recommending for their integration with Jaeger agents. I'm just not sure why one method via the driver code works but agent.tags doesn't for the same content.

[Aaronontheweb]

Here's the relevant comment from the Elastic thread I mentioned: elastic/apm#212 (comment)

[axw]

@Aaronontheweb I think you need $(ELASTIC_APM_TOKEN) (parentheses) rather than ${ELASTIC_APM_TOKEN} (curly braces) in your k8s yaml. If that doesn't fix it, please head over to https://discuss.elastic.co/c/observability/apm/58 and we can help you out there. I don't think this is a problem with Jaeger.

@yurishkuro I think the use of process tags for this is not great. We needed a way to have jaeger-agent send auth tokens and couldn't find a better way. If you have any recommendations I'd be keen to hear them.

Would you be open to a PR that adds an option to send gRPC request metadata? Something like --reporter.grpc.headers "key=value", which would result in key=value being added to outgoing metadata. I think this could also be useful when sending to opentelemetry-collector.

[yurishkuro]

sgtm. we already have a similar flag to read bearer token from file when talking to ES

[Aaronontheweb]

@Aaronontheweb I think you need $(ELASTIC_APM_TOKEN) (parentheses) rather than ${ELASTIC_APM_TOKEN} (curly braces) in your k8s yaml. If that doesn't fix it, please head over to https://discuss.elastic.co/c/observability/apm/58 and we can help you out there. I don't think this is a problem with Jaeger.

FWIW I did try that as well; same outcome. I'll report an issue on the Elastic APM tracker.

[pentago]

any news on this issue?
Having the same problems sending traces over RPC to Elastic APM..

[axw]

@pentago no news. Please open a topic at https://discuss.elastic.co/c/observability/apm/58.