-
Notifications
You must be signed in to change notification settings - Fork 792
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can not execute the operation on ended Span
/ tls.connect .. You can only call end() on a span once.
#3999
Comments
This seems like it might be a bug in one of the instrumentations. Is the telemetry incorrect or are you just seeing unexpected logs? |
Those 3 messages are most definitely unexpected. And very predictable - "just start and wait 2 minutes" (or so). All other telemetry looks correct - despite the above 3 log messages. |
We are seeing the same issue when using |
@stdedos What version of Node.js are you running? Since I cannot see this information in your original post. I am also experiencing this on Node.js 20 similarly when using |
Answering for my teammate @stdedos, we are seeing this on node 18 |
@googol Ok so I guess it's not neccessarily node version related. Doing a little digging i found a change in This has landed in Maybe if you have time you can try and install exactly |
No luck: u@h:prj$ npm i @opentelemetry/[email protected]
added 1 package, changed 2 packages, and audited 1113 packages in 15s
u@h:prj$ npm run dev
> [email protected] dev
> nodemon
[nodemon] 2.0.22
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): src/**/*
[nodemon] watching extensions: ts
[nodemon] starting `ts-node -r ./src/initialize-telemetry.ts --transpile-only ./src/index.ts`
...
{
"level": "info",
"message": "Server listening for connections on 0.0.0.0:4900",
"timestamp": "2023-08-09T15:02:46.259Z"
}
{
"level": "warn",
"message": "Can not execute the operation on ended Span {traceId: b67fc07d0cf9496fe8dad2a49c0dc4f0, spanId: cac5b97cca3baf39}",
"span_id": "8581abcc0d119d2f",
"timestamp": "2023-08-09T15:05:50.430Z",
"trace_flags": "01",
"trace_id": "b67fc07d0cf9496fe8dad2a49c0dc4f0"
}
{
"level": "warn",
"message": "Can not execute the operation on ended Span {traceId: b67fc07d0cf9496fe8dad2a49c0dc4f0, spanId: cac5b97cca3baf39}",
"span_id": "8581abcc0d119d2f",
"timestamp": "2023-08-09T15:05:50.430Z",
"trace_flags": "01",
"trace_id": "b67fc07d0cf9496fe8dad2a49c0dc4f0"
}
{
"level": "error",
"message": "tls.connect b67fc07d0cf9496fe8dad2a49c0dc4f0-cac5b97cca3baf39 - You can only call end() on a span once.",
"span_id": "8581abcc0d119d2f",
"timestamp": "2023-08-09T15:05:50.431Z",
"trace_flags": "01",
"trace_id": "b67fc07d0cf9496fe8dad2a49c0dc4f0"
} |
Hello everyone! Any progress? 🙃 |
@stdedos Do you have an example of such a span? Would be interesting to see whether it had any relevant attributes (e.g. errors). |
@seemk, unfortunately, I cannot provide a direct example of such a span due to the private nature of the project. Doesn't my example replicate locally? |
We are also seeing them on every |
I haven't managed to reproduce it yet, but my hunch is that a TLS socket error is emitted after a successful |
If possible, what's the library you are doing the outgoing requests with? And which HTTP server are you using? Trying to replicate it as close as possible. |
Server is Client is "axios": "^1.4.0", |
It seems that import { NetInstrumentation } from '@opentelemetry/instrumentation-net' Is the culprit in our situation. |
Registered via: function main() {
const resource = createResource()
const provider = new NodeTracerProvider({
resource,
})
const exporter = createExporter()
const spanProcessor = createSpanProcessor(exporter)
provider.addSpanProcessor(spanProcessor)
provider.register()
registerInstrumentations({
instrumentations: [
// new ...({
// ...
// }),
// new ...({
// ...
// }),
// new ...(),
// new ...(),
new NetInstrumentation(),
// ...(),
],
tracerProvider: provider,
})
const exitHandler = () => {
console.log('Shutting down OpenTelemetry...')
// Schedule the shutdown to happen after the current event loop iteration
// completes. This is necessary because we want to give the exporter a
// chance to send any remaining spans.
provider
.shutdown()
.then(() => console.log('OpenTelemetry shutdown complete'))
.catch((ex) => console.error('OpenTelemetry shutdown failed', ex))
}
process.on('SIGINT', exitHandler)
process.on('SIGTERM', exitHandler)
} |
Hi all, looks like this is caused by the net instrumentation, which is part of the contrib repo. There's an issue there for the same problem. I'll close this issue here as a duplicate but you can follow the progress over at open-telemetry/opentelemetry-js-contrib#1775 If you think that's not correct, please feel free to reach out and I'll re-open this issue and transfer it to the contrib repo. |
What happened?
Steps to Reproduce
Run a private repo using
nodemon
Expected Result
Actual Result
logs
Additional Details
Seems related to:
OpenTelemetry Setup Code
package.json
Relevant log output
The text was updated successfully, but these errors were encountered: