Suppressing autoinstrumentation properties do not have effect

[greatvovan]

Expected behavior

The official documentation describes a number of parameters to suppress automatic instrumentation in entirety or for specific libraries. It is fair to expect that instrumenting with Microsoft's agent will follow the same logic and obey the same configuration parameters.

Actual behavior

All suppression properties are ignored except otel.javaagent.enabled.

To Reproduce

Originally faced this problem in a Databricks cluster, where alternative instrumentation approaches don't work either. I assumed that it might be an incompatibility with Spark runtime or Databricks environment, so I decided to test it on a smaller program, but it confirmed my fears.

The code: microsoft/ApplicationInsights-Java-Repros#13

% docker run -it --rm -v $(pwd)/ApplicationInsights-Java-Repros/SuppresAutoinstrumentation:/prj maven bash
# root@bc3760e5405c:/# cd /prj
# mkdir /dl
# curl -sLOJ https:/microsoft/ApplicationInsights-Java/releases/download/3.6.1/applicationinsights-agent-3.6.1.jar --output-dir /dl
# mvn package
...
# export APPLICATIONINSIGHTS_CONNECTION_STRING="..."
# # java -Dotel.instrumentation.common.default-enabled=false -javaagent:/dl/applicationinsights-agent-3.6.1.jar -jar target/my-project-1.0-SNAPSHOT.jar
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
2024-10-19 20:31:14.701Z INFO  c.m.a.a.i.c.ConfigurationBuilder - Some telemetry may be sampled out because a default sampling configuration was added in version 3.4.0 to reduce the default billing cost. You can set the sampling configuration explicitly: https://learn.microsoft.com/azure/azure-monitor/app/java-standalone-config#sampling
2024-10-19 20:31:16.744Z INFO  c.m.applicationinsights.agent - Application Insights Java Agent 3.6.1 started successfully (PID 160, JVM running for 2.861 s)
2024-10-19 20:31:16.744Z INFO  c.m.applicationinsights.agent - Java version: 21.0.4, vendor: Eclipse Adoptium, home: /opt/java/openjdk
2024-10-19 20:31:17 WARN  [main] root - Hello Azure
2024-10-19 20:31:18 INFO  [main] root - Log from a trace

Check your Application Insights instance to ensure that the logs are coming, contrary to the suppression parameter.
EXPECTED: Only the trace is coming as it was sent through OpenTelemetry API.

Let's compare this behavior with the vanilla agent from OpenTelemetry.

# curl -sLOJ https:/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.9.0/opentelemetry-javaagent.jar --output-dir /dl
# export OTEL_LOGS_EXPORTER=console OTEL_METRICS_EXPORTER=none OTEL_TRACES_EXPORTER=none
# java -javaagent:/dl/opentelemetry-javaagent.jar -jar target/my-project-1.0-SNAPSHOT.jar
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2024-10-19 20:43:08:770 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.9.0
2024-10-19T20:43:10.999Z WARN 'Hello Azure' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: root:] {}
2024-10-19 20:43:11 WARN  [main] root - Hello Azure
2024-10-19T20:43:12.046Z INFO 'Log from a trace' : b8faff3761f8c71bba023991e6b7cb26 08f4ec2b7bf7b21b [scopeInfo: root:] {}
2024-10-19 20:43:12 INFO  [main] root - Log from a trace

Note the log records printed by the Console Exporter.
Now let's disable the default instrumentation:

# java -Dotel.instrumentation.common.default-enabled=false -javaagent:/dl/opentelemetry-javaagent.jar -jar target/my-project-1.0-SNAPSHOT.jar
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
[otel.javaagent 2024-10-19 20:45:51:305 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 2.9.0
2024-10-19 20:45:52 WARN  [main] root - Hello Azure
2024-10-19 20:45:53 INFO  [main] root - Log from a trace

Console Exporter does not print any more.

Targeted suppression, like otel.instrumentation.log4j-appender.enabled=false works in the same way, disabling instrumentation in the vanilla agent, but having no effect with Microsoft's agent.

# java -Dotel.instrumentation.log4j-appender.enabled=false -javaagent:/dl/applicationinsights-agent-3.6.1.jar -jar target/my-project-1.0-SNAPSHOT.jar
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
2024-10-19 20:55:00.879Z INFO  c.m.a.a.i.c.ConfigurationBuilder - Some telemetry may be sampled out because a default sampling configuration was added in version 3.4.0 to reduce the default billing cost. You can set the sampling configuration explicitly: https://learn.microsoft.com/azure/azure-monitor/app/java-standalone-config#sampling
2024-10-19 20:55:02.888Z INFO  c.m.applicationinsights.agent - Application Insights Java Agent 3.6.1 started successfully (PID 336, JVM running for 2.804 s)
2024-10-19 20:55:02.889Z INFO  c.m.applicationinsights.agent - Java version: 21.0.4, vendor: Eclipse Adoptium, home: /opt/java/openjdk
2024-10-19T20:55:03.488Z WARN 'Hello Azure' : 00000000000000000000000000000000 0000000000000000 [scopeInfo: root:] {thread.id=1, thread.name="main"}
2024-10-19 20:55:03 WARN  [main] root - Hello Azure
2024-10-19T20:55:04.538Z INFO 'Log from a trace' : 8cae2db0a04ff7f4bffc047bf1d02c4c 9424ef32b83aeda9 [scopeInfo: root:] {applicationinsights.internal.operation_name="My-span", thread.id=1, thread.name="main"}
2024-10-19 20:55:04 INFO  [main] root - Log from a trace

The same applies to all other kinds of telemetry and other libraries.

Now in Databricks clusters I am blocked with telemetry on all fronts. The default autoinstrumentation generates insane amounts of telemetry from all imaginable Spark internals and makes instrumentation irrational due to ingestion costs. With Azure Exporter for Java just nothing works.

System information

Please provide the following information:

• SDK Version: applicationinsights-agent-3.6.1.jar, OpenTelemetry SDK 1.43.0 (BOM v. 2.9.0)
• OS type and version: any OS (observed on MacOS, Linux container, Linux-like VM)
• Application Server type and version (if applicable): N/A
• Using spring-boot? No.
• Additional relevant libraries (with version, if applicable): N/A

Logs

Maintainers, please note that the link provided by the template (Turn on SDK logs – https://docs.microsoft.com/en-us/azure/application-insights/app-insights-java-troubleshoot#debug-data-from-the-sdk) is no longer valid. The alternative document I found says that the logging is enabled by default into the text file, which I am attaching below. If more logging is required, please provide guidance.

2024-10-19 20:31:14.701Z INFO  c.m.a.a.i.c.ConfigurationBuilder - Some telemetry may be sampled out because a default sampling configuration was added in version 3.4.0 to reduce the default billing cost. You can set the sampling configuration explicitly: https://learn.microsoft.com/azure/azure-monitor/app/java-standalone-config#sampling
2024-10-19 20:31:16.744Z INFO  c.m.applicationinsights.agent - Application Insights Java Agent 3.6.1 started successfully (PID 160, JVM running for 2.861 s)
2024-10-19 20:31:16.744Z INFO  c.m.applicationinsights.agent - Java version: 21.0.4, vendor: Eclipse Adoptium, home: /opt/java/openjdk