-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
spanmetrics connector generating extreme grpc traffic #20306
Comments
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
@devrimdemiroz The reason is #19216, |
@kovrus, I truly appreciate your quick response! If you could provide me with a little bit more on the transform processor configuration I need to add, you'll be an absolute time-saver for me. Thanks in advance! |
@devrimdemiroz something like that will reduce the number of resource scopes to the number of the services that produce telemetry. If we want to allow the old behavior, one resource scope for everything, we should wrap #19467 up. ...
processors:
transform:
trace_statements:
- context: resource
statements:
- keep_keys(attributes, ["service.name"])
...
service:
pipelines:
traces/spanmetrics:
receivers: [otlp]
processors: [transform]
exporters: [spanmetrics]
metrics/spanmetrics:
receivers: [spanmetrics]
exporters: [prometheus]
... |
@kovrus, thank you for sharing the precise configuration; it works perfectly. However, I'm unsure if it's absolutely necessary or not. My goal is to create a more straightforward and comprehensible configuration using the new connector config. To achieve this, I've had to add a layer that I haven't used or been familiar with before, which wasn't required by the previous processor. I'm not questioning the importance or potential benefits it may offer; I'm merely curious about the rationale behind some extra lines that aren't immediately clear. Nevertheless, I would recommend including it as part of the default spanmetrics connector config in the documentation. Since the transform config works, I'll consider this matter resolved. Thanks for your time. |
@devrimdemiroz yes, we should add a more comprehensive readme for the span metrics connector and its differences from the processor. I've tried to call out that more metrics will be generated when using the connector here, but we probably should provide a better explanation. The transform processor with @djaglowski I think, we should probably revisit #19467 and allow users to control what attributes are going to be added to generated metrics resource scopes. Maybe, by default, we can keep resource |
My only concern is that we may find ourselves needing to add more and more "transform" capabilities to this connector as well as others. However, if emitting consolidated metrics based on resource attributes appears to be a particularly common case, then I support it. |
Component(s)
connector/spanmetrics
What happened?
Description
I replaced the spanmetrics processor config on opentelemetry demo app with the new spanmetrics connector. The otlp grpc receiver observed traffic increased almost 10,000 times. Accordingly calls (previously calls_total) and related spanmetrics also linearly explode. See the screenshots at the bottom.
Steps to Reproduce
Following configuration is used in replacement for spanmetrics processor:
Expected Result
The expected result is to be inline with spanmetrics processor runs.
When processor runs:
Actual Result
When connector runs:
Collector version
0.74.0
Environment information
Environment
Images
IMAGE_VERSION=1.3.1
IMAGE_NAME=ghcr.io/open-telemetry/demo
OpenTelemetry Collector configuration
Log output
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: