AgentWriter exception became visible in 2.11.0 version

[doctornkz-intelas]

Summary of the problem

Recently we upgraded ddtrace package and now we see Connection errors on several occasions, such as logging, pytest, etc.
It might be a duplicate of the issue, also somehow related to this issue if I am not mistaken.

Which version of dd-trace-py are you using?

2.12.2, but the issue became visible in the >=2.11.0 version.

Which version of pip are you using?

23.0.1

Which libraries and their versions are you using?

`pip freeze`

bytecode==0.15.1 ddtrace==2.12.2 Deprecated==1.2.14 envier==0.5.2 importlib_metadata==8.4.0 opentelemetry-api==1.27.0 protobuf==5.28.2 typing_extensions==4.12.2 wrapt==1.16.0 xmltodict==0.13.0 zipp==3.20.2

How can we reproduce your problem?

docker run -it python:3.10 bash
root@dc88ca0512aa:/# pip install ddtrace==2.12.2
[SKIPPED INSTALLATION LOGS]
root@dc88ca0512aa:/# python
Python 3.10.15 (main, Sep 12 2024, 21:09:01) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import logging
>>> logging.getLogger().level = logging.DEBUG
>>> import ddtrace
git tags from env:
git tags:
initialized RateSampler, sample 100% of traces
initialized DatadogSampler(agent_rates={}, limiter=RateLimiter(rate_limit=100, tokens=100, last_update_ns=9283641597371532, effective_rate=1.0), rules=[])
initialized RateSampler, sample 100% of traces
initialized DatadogSampler(agent_rates={}, limiter=RateLimiter(rate_limit=100, tokens=100, last_update_ns=9283641597534341, effective_rate=1.0), rules=[])
Statsd buffering is disabled
Statsd periodic buffer flush is disabled
initialized trace processor <ddtrace._trace.processor.TraceSamplingProcessor object at 0x7f696a693400>
initialized trace processor <ddtrace._trace.processor.TraceTagsProcessor object at 0x7f696a693370>
initialized processor <ddtrace._trace.processor.TopLevelSpanProcessor object at 0x7f696a693340>
initialized processor SpanAggregator(True, 300, [<ddtrace.internal.peer_service.processor.PeerServiceProcessor object at 0x7f696a6934c0>, <ddtrace.internal.schema.processor.BaseServiceProcessor object at 0x7f696a6934f0>, <ddtrace._trace.processor.TraceSamplingProcessor object at 0x7f696a693400>, <ddtrace._trace.processor.TraceTagsProcessor object at 0x7f696a693370>], AgentWriter(status=<ServiceStatus.STOPPED: 'stopped'>, intake_url='http://localhost:8126', dogstatsd=<ddtrace.vendor.dogstatsd.base.DogStatsd object at 0x7f696a6937f0>))
>>> failed to send telemetry to http://localhost:8126/telemetry/proxy/api/v2/apmtelemetry.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/telemetry/writer.py", line 168, in send_event
    conn.request("POST", self._endpoint, rb_json, headers)
  File "/usr/local/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/local/lib/python3.10/http/client.py", line 942, in connect
    self.sock = self._create_connection(
  File "/usr/local/lib/python3.10/socket.py", line 857, in create_connection
    raise err
  File "/usr/local/lib/python3.10/socket.py", line 845, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

What is the result that you get?

I am receiving annoying errors about ddtrace itself.

>>> failed to send telemetry to http://localhost:8126/telemetry/proxy/api/v2/apmtelemetry.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/telemetry/writer.py", line 168, in send_event
    conn.request("POST", self._endpoint, rb_json, headers)
  File "/usr/local/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/local/lib/python3.10/http/client.py", line 942, in connect
    self.sock = self._create_connection(
  File "/usr/local/lib/python3.10/socket.py", line 857, in create_connection
    raise err
  File "/usr/local/lib/python3.10/socket.py", line 845, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
failed to send telemetry to http://localhost:8126/telemetry/proxy/api/v2/apmtelemetry.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/ddtrace/internal/telemetry/writer.py", line 168, in send_event
    conn.request("POST", self._endpoint, rb_json, headers)
  File "/usr/local/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/usr/local/lib/python3.10/http/client.py", line 942, in connect
    self.sock = self._create_connection(
  File "/usr/local/lib/python3.10/socket.py", line 857, in create_connection
    raise err
  File "/usr/local/lib/python3.10/socket.py", line 845, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

What is the result that you expected?

I am expecting no errors in my Python runtime. Only if I am explicitly using DD_TRACE_DEBUG=true. Together with the DD_INSTRUMENTATION_TELEMETRY_ENABLED parameter the ddtrace is creating too much noise during debugging and breaking the development experience.

Workaround and additional information

I can mitigate the issue like this:

DD_TRACE_DEBUG=0 DD_INSTRUMENTATION_TELEMETRY_ENABLED=false python
Python 3.10.15 (main, Sep 12 2024, 21:09:01) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import logging
>>> logging.getLogger().level = logging.DEBUG
>>> import ddtrace
git tags from env:
git tags:
initialized RateSampler, sample 100% of traces
initialized DatadogSampler(agent_rates={}, limiter=RateLimiter(rate_limit=100, tokens=100, last_update_ns=9280871996580493, effective_rate=1.0), rules=[])
initialized RateSampler, sample 100% of traces
initialized DatadogSampler(agent_rates={}, limiter=RateLimiter(rate_limit=100, tokens=100, last_update_ns=9280871997063324, effective_rate=1.0), rules=[])
Statsd buffering is disabled
Statsd periodic buffer flush is disabled
initialized trace processor <ddtrace._trace.processor.TraceSamplingProcessor object at 0x7fdd09c31780>
initialized trace processor <ddtrace._trace.processor.TraceTagsProcessor object at 0x7fdd09c31720>
initialized processor <ddtrace._trace.processor.TopLevelSpanProcessor object at 0x7fdd09c316f0>
initialized processor SpanAggregator(True, 300, [<ddtrace.internal.peer_service.processor.PeerServiceProcessor object at 0x7fdd09c31840>, <ddtrace.internal.schema.processor.BaseServiceProcessor object at 0x7fdd09c31870>, <ddtrace._trace.processor.TraceSamplingProcessor object at 0x7fdd09c31780>, <ddtrace._trace.processor.TraceTagsProcessor object at 0x7fdd09c31720>], AgentWriter(status=<ServiceStatus.STOPPED: 'stopped'>, intake_url='http://localhost:8126/', dogstatsd=<ddtrace.vendor.dogstatsd.base.DogStatsd object at 0x7fdd09c31a50>))
>>>

Commit where this behavior was changed is here, if I am not mistaken.