OpenTelemetry exporter failed to initialize with exception inets_not_started

[IwoHerka]

I'm trying to deploy Phoenix application with OpenTelemetry tracing in a Docker container. Dockerfile I was using so far (sucessfully) is a typical one (i.e. generated by mix phx.gen.release --docker). Following versions are used:

ARG ELIXIR_VERSION=1.15.4
ARG OTP_VERSION=26.0.2
ARG DEBIAN_VERSION=bullseye-20230612-slim

So, the image is hexpm/elixir:1.15.4-erlang-26.0.2-debian-bullseye-20230612-slim.
(I'm running the same versions locally for development, the only difference I'm working on Ubuntu 22.04 via WSL2.)
(But I've confirmed the same happens on Ubuntu image)

The problem is that then running this container, a following message is produced which indicates that OTLP exporter fails (with MIX_ENV=prod):

=WARNING REPORT==== 6-Sep-2023::20:15:42.868641 ===
OTLP exporter failed to initialize with exception error:{badmatch,
                                                          {error,
                                                           inets_not_started}}
=WARNING REPORT==== 6-Sep-2023::20:15:42.880076 ===
Failed to load OTP-trusted CAs: error:{badmatch,{error,enoent}}, falling back to hardcoded authorities

This doesn't happen when running code in (both in dev and prod) locally, outside container.

My OTLP config is almost the same as in https://opentelemetry.io/docs/instrumentation/erlang/getting-started/, namely:

mix.exs:

# (in this order):

{:opentelemetry_exporter, "~> 1.6"},
{:opentelemetry, "~> 1.3"},
{:opentelemetry_api, "~> 1.2"},
{:opentelemetry_phoenix, "~> 1.1"},
{:opentelemetry_cowboy, "~> 0.2"},
{:opentelemetry_ecto, "~> 1.1"},

# and:

defp releases() do
  [
    foo: [
      applications: [
        opentelemetry: :temporary,
        opentelemetry_exporter: :permanent
      ]
    ]
  ]
end

application.ex:

:opentelemetry_cowboy.setup()
OpentelemetryPhoenix.setup(adapter: :cowboy2)
OpentelemetryEcto.setup([:foo, :repo])

config/runtime.ex:

  otlp_protocol = System.get_env("OTEL_PROTOCOL") |> String.to_atom()
  otlp_endpoint = System.get_env("OTEL_ENDPOINT")

  config :opentelemetry,
    span_processor: :batch,
    traces_exporter: :otlp

  config :opentelemetry_exporter,
    otlp_protocol: otlp_protocol,
    otlp_endpoint: otlp_endpoint

  config :opentelemetry, :processors, [
    {:otel_simple_processor, %{}}
  ]

I've figured out that the problem only occurs in a release, that's why I only see it in a container, not when starting via mix phx.server or iex -S mix phx.server. | I've figured out that the problem only occurs in a release, that's why I only see it in a container, not when starting via mix phx.server or iex -S mix phx.server.

When run with lower log level I also get:

[debug] OTLP exporter failed to initialize: exception error: no match of right hand side value {error,inets_not_started}
  in function  opentelemetry_exporter:start_httpc/1 (/home/poly/P/glimpse-api/deps/opentelemetry_exporter/src/opentelemetry_exporter.erl, line 239)
  in call from opentelemetry_exporter:init/1 (/home/poly/P/glimpse-api/deps/opentelemetry_exporter/src/opentelemetry_exporter.erl, line 210)
  in call from otel_exporter:init/1 (/home/poly/P/glimpse-api/deps/opentelemetry/src/otel_exporter.erl, line 44)
  in call from otel_batch_processor:init_exporter/2 (/home/poly/P/glimpse-api/deps/opentelemetry/src/otel_batch_processor.erl, line 317)
  in call from otel_batch_processor:idle/3 (/home/poly/P/glimpse-api/deps/opentelemetry/src/otel_batch_processor.erl, line 197)
  in call from gen_statem:loop_state_callback/11 (gen_statem.erl, line 1377)
  in call from proc_lib:init_p_do_apply/3 (proc_lib.erl, line 241)

Does anyone have any idea why this problem occurs?

Thanks!

[tsloughter]

Hm, I wonder if its the lack of CA's.. but the fallback says it is happening. I thought this was resolved with #547 but maybe its not fully solved.

[IwoHerka]

I can confirm that changing the order of opentelemetry and opentelemetry_exporter under releases in mix.exs fixes the issue (exporter must be first):

foo: [
  applications: [
    opentelemetry_exporter: :permanent,
    opentelemetry: :temporary
  ]
]

On startup, the following message is still produced, but the exporter seems to be working:

=WARNING REPORT==== 7-Sep-2023::09:30:45.040687 ===
Failed to load OTP-trusted CAs: error:{badmatch,{error,enoent}}, falling back to hardcoded authorities

[tsloughter]

Well that makes me scratch my head even harder... :)

[IwoHerka]

(I posted the previous sample, the exporter is first after the fix)

[akramhussein]

Having same issue, with the following setup:

Dependencies:

    [
      {:opentelemetry_api, "~> 1.2.2"},
      {:opentelemetry_cowboy, "~> 0.2.1"},
      {:opentelemetry_ecto, "~> 1.2.0"},
      {:opentelemetry_exporter, "~> 1.6.0"},
      {:opentelemetry_liveview, "~> 1.0.0-rc.4"},
      {:opentelemetry_oban, "~> 1.0.0"},
      {:opentelemetry_phoenix, "~> 1.2.0"},
      {:opentelemetry_tesla, "~> 2.3.0"},
      {:opentelemetry_semantic_conventions, "~> 0.2.0"},
      {:opentelemetry, "~> 1.3.1"}
    ]

Using:

elixir 1.15.7-otp-26
erlang 26.2.2

In a Docker image using docker.io/hexpm/elixir:1.15.7-erlang-26.2.2-debian-bullseye-20240130-slim

Have tried the following:

1. apt-get ca-certificates and then run update-ca-certificates --fresh
2. Both ways of ordering applications: [opentelemetry: :temporary, opentelemetry_exporter: :permanent]
3. Adding to mix: {:tls_certificate_check, "~> 1.21.0", override: true},

Still receiving the same message.

[danibachar]

We are facing the same error - any workarounds thought about it?

15:46:26.213 [debug] OTLP exporter failed to initialize: exception error: no match of right hand side value {error,inets_not_started}
  in function  opentelemetry_exporter:start_httpc/1 (/app/deps/opentelemetry_exporter/src/opentelemetry_exporter.erl, line 239)
  in call from opentelemetry_exporter:init/1 (/app/deps/opentelemetry_exporter/src/opentelemetry_exporter.erl, line 210)
  in call from otel_exporter:init/1 (/app/deps/opentelemetry/src/otel_exporter.erl, line 44)
  in call from otel_batch_processor:init_exporter/2 (/app/deps/opentelemetry/src/otel_batch_processor.erl, line 327)
  in call from otel_batch_processor:idle/3 (/app/deps/opentelemetry/src/otel_batch_processor.erl, line 200)
  in call from gen_statem:loop_state_callback/11 (gen_statem.erl, line 3113)
  in call from proc_lib:init_p_do_apply/3 (proc_lib.erl, line 329)
15:46:26.214 [warning] OTLP exporter failed to initialize with exception :error:{:badmatch, {:error, :inets_not_started}}
15:46:26.262 [info] Loading 129 CA(s) from :otp store
{exit,terminating,[{application_controller,call,2,[{file,"application_controller.erl"},{line,511}]},{application,enqueue_or_start,6,[{file,"application.erl"},{line,380}]},{application,ensure_all_started,3,[{file,"application.erl"},{line,359}]},{elixir,start_cli,0,[{file,"src/elixir.erl"},{line,195}]},{init,start_it,1,[]},{init,start_em,1,[]},{init,do_boot,3,[]}]}

[bryannaegele]

Did you try this? It's also best if you list the exporter dep before opentelemetry in you dependencies. Ordering does matter.

#625 (reply in thread)

[danibachar]

Thanks for the prompt response,.

Yes, unfortunately it does not work...
Let me grab the dependencies and docker version...