Trace agent fails to start on arm64 nodes #6635
Comments
|
Hi @dan-lind 👋 From the stacktrace you shared, it looks like the native library in charge of UDS is not able to load due to a missing Can you tell us on which environment your traced application is running on? |
|
Hi! We run our containers on amazoncorretto:17/21, which bulds on amazonlinux:2, that in turn indeed builds on alpine:3.17. Here's the output, look like the agent expects a newer version than what we have? |
It looks like Moreover, I checked if there is a way to upgrade to a newer version but it seems you can't according to Amazon. Instead, you would have to use an image based on a more recent I checked So I would recommend to switch to |
|
Oh, interesting! Is this version requirement something that has changed in a recent tracing agent version, or would it be related to us running arm nodes? I will try with amazoncorretto:17-al2023 and get back |
|
This is something related to our About the different behavior according the architecture, I tried to load and link the library in both amd64 and arm64 docker images: Using amd64
bash-4.2# uname -a
Linux 9ba5e0f9b50f 6.6.12-linuxkit #1 SMP Fri Jan 19 08:53:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
bash-4.2# ldd libjffi-1.2.so
libc.so.6 => /lib64/libc.so.6 (0x00007fffff00d000)
/lib64/ld-linux-x86-64.so.2 (0x00007fffffdda000)Using arm64
bash-4.2# uname -a
Linux 75e832fe605c 6.6.12-linuxkit #1 SMP Fri Jan 19 08:53:17 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
bash-4.2# ldd libjffi-1.2.so
./libjffi-1.2.so: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by ./libjffi-1.2.so)
linux-vdso.so.1 (0x0000ffffa177d000)
libc.so.6 => /lib64/libc.so.6 (0x0000ffffa1588000)
/lib/ld-linux-aarch64.so.1 (0x0000ffffa173f000)You can see it failing on arm64 even if it succeed on amd64 with glibc 2.26. |
|
Thanks for a great explanation and for your help! I tried running with |
|
You're welcome! We will also give a shot at fixing the issue upstream, in the |
|
FYI: the JNR/JFFI project have released a fix for this which will be in the next release of the Java tracer, 1.31.0 |
|
🤖 This issue has been addressed in the latest release. See full details in the Release Notes. |
We have been running with the java dd-trace-agent on amd64 nodes for a couple of years, without any issues. We recently started running our containers on a combination of amd64 and arm64 nodes. We are now seeing that the agent fails to start properly on the arm64 nodes.
We see very long java stack traces from the agent, and since the agent doesn't seem to start properly, we get one row from the stack trace as a separate log record, rather than as a single stack trace.
I found this issues which looks similar and mentions a fix was release, #4702, but I guess this might be a different issue
We are running with dd-trace-agent 1.29, datadog-agent and cluster-agent 7.50.3. Below is an example of the stack-trace.
The text was updated successfully, but these errors were encountered: