Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aarch64 builds using nightly fail with missing getauxval in init_have_lse_atomics in cpu_model.c in compiler_builtins #598

Closed
tonyg opened this issue Oct 8, 2021 · 9 comments
Labels
A-arm Area: ARM targets bug upstream

Comments

@tonyg
Copy link

tonyg commented Oct 8, 2021

[Updated] Workaround

See this comment.

To reproduce

  1. mkdir foo
  2. cd foo
  3. cargo init
  4. cross build --target=aarch64-unknown-linux-musl --release --all-targets -- note that this succeeds
  5. echo 'nightly' > rust-toolchain
  6. cross build --target=aarch64-unknown-linux-musl --release --all-targets -- note that this fails

Actual outcome

Step 6 fails with

$ cross build --target=aarch64-unknown-linux-musl --release --all-targets
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'

  nightly-x86_64-unknown-linux-gnu unchanged - rustc 1.57.0-nightly (485ced56b 2021-10-07)

info: checking for self-updates
   Compiling foo v0.1.0 (/project)
error: linking with `aarch64-linux-musl-gcc` failed: exit status: 1
  |
  = note: "aarch64-linux-musl-gcc" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crt1.o" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crti.o" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtbegin.o" "/target/aarch64-unknown-linux-musl/release/deps/foo-a9545cdbbb8683ea.foo.4b08363c-cgu.0.rcgu.o" "/target/aarch64-unknown-linux-musl/release/deps/foo-a9545cdbbb8683ea.foo.4b08363c-cgu.1.rcgu.o" "/target/aarch64-unknown-linux-musl/release/deps/foo-a9545cdbbb8683ea.foo.4b08363c-cgu.2.rcgu.o" "/target/aarch64-unknown-linux-musl/release/deps/foo-a9545cdbbb8683ea.6er9z6ghb47uqlg.rcgu.o" "-Wl,--as-needed" "-L" "/target/aarch64-unknown-linux-musl/release/deps" "-L" "/target/release/deps" "-L" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib" "-Wl,--start-group" "-Wl,-Bstatic" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd-c44788b657ddddd5.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libpanic_unwind-2becdccda730ce42.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libminiz_oxide-3a7898da562cf313.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libadler-fc70ddb33855e2e8.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libobject-5fb8217618711e88.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libmemchr-c061be432db3e13d.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libaddr2line-5a6d99ea33f6ceae.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libgimli-6db2c8017781e41d.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd_detect-7d0d2f13c64acb61.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/librustc_demangle-77c2cc6c5af8536c.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libhashbrown-8ca99a806b6c7500.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/librustc_std_workspace_alloc-eced44336e84fd45.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libunwind-ec4fe46000190b79.rlib" "-lunwind" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libcfg_if-692ed2e2b4273428.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/liblibc-72b57ab90668a509.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/liballoc-7aed2b0a9d39bb09.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/librustc_std_workspace_core-e0db88e40d9c7e0b.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libcore-fcedc0d4b8cb02ca.rlib" "-Wl,--end-group" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libcompiler_builtins-cbf6a032efba8222.rlib" "-Wl,-Bdynamic" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-nostartfiles" "-L" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib" "-L" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained" "-o" "/target/aarch64-unknown-linux-musl/release/deps/foo-a9545cdbbb8683ea" "-Wl,--gc-sections" "-static" "-no-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-lgcc" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtend.o" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtn.o"
  = note: /rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libcompiler_builtins-cbf6a032efba8222.rlib(cpu_model.o): In function `init_have_lse_atomics':
          /cargo/registry/src/github.com-1ecc6299db9ec823/compiler_builtins-0.1.49/./lib/builtins/cpu_model.c:786: undefined reference to `getauxval'
          collect2: error: ld returned 1 exit status

  = help: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

error: could not compile `foo` due to previous error
warning: build failed, waiting for other jobs to finish...
error: linking with `aarch64-linux-musl-gcc` failed: exit status: 1
  |
  = note: "aarch64-linux-musl-gcc" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crt1.o" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crti.o" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtbegin.o" "/target/aarch64-unknown-linux-musl/release/deps/foo-82633e12cd2e5a19.foo.b81dabbf-cgu.0.rcgu.o" "/target/aarch64-unknown-linux-musl/release/deps/foo-82633e12cd2e5a19.foo.b81dabbf-cgu.1.rcgu.o" "/target/aarch64-unknown-linux-musl/release/deps/foo-82633e12cd2e5a19.foo.b81dabbf-cgu.2.rcgu.o" "/target/aarch64-unknown-linux-musl/release/deps/foo-82633e12cd2e5a19.11e6h28jheunbtcu.rcgu.o" "-Wl,--as-needed" "-L" "/target/aarch64-unknown-linux-musl/release/deps" "-L" "/target/release/deps" "-L" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib" "-Wl,-Bstatic" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libtest-3a3779ef37dcf104.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libgetopts-4b9c52f1a6321a32.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libunicode_width-3a27e1f0db71671d.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/librustc_std_workspace_std-9ce51e2ad117da5c.rlib" "-Wl,--start-group" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd-c44788b657ddddd5.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libpanic_unwind-2becdccda730ce42.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libminiz_oxide-3a7898da562cf313.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libadler-fc70ddb33855e2e8.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libobject-5fb8217618711e88.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libmemchr-c061be432db3e13d.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libaddr2line-5a6d99ea33f6ceae.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libgimli-6db2c8017781e41d.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libstd_detect-7d0d2f13c64acb61.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/librustc_demangle-77c2cc6c5af8536c.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libhashbrown-8ca99a806b6c7500.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/librustc_std_workspace_alloc-eced44336e84fd45.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libunwind-ec4fe46000190b79.rlib" "-lunwind" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libcfg_if-692ed2e2b4273428.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/liblibc-72b57ab90668a509.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/liballoc-7aed2b0a9d39bb09.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/librustc_std_workspace_core-e0db88e40d9c7e0b.rlib" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libcore-fcedc0d4b8cb02ca.rlib" "-Wl,--end-group" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libcompiler_builtins-cbf6a032efba8222.rlib" "-Wl,-Bdynamic" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-nostartfiles" "-L" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib" "-L" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained" "-o" "/target/aarch64-unknown-linux-musl/release/deps/foo-82633e12cd2e5a19" "-Wl,--gc-sections" "-static" "-no-pie" "-Wl,-zrelro" "-Wl,-znow" "-Wl,-O1" "-nodefaultlibs" "-lgcc" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtend.o" "/rust/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/crtn.o"
  = note: /rust/lib/rustlib/aarch64-unknown-linux-musl/lib/libcompiler_builtins-cbf6a032efba8222.rlib(cpu_model.o): In function `init_have_lse_atomics':
          /cargo/registry/src/github.com-1ecc6299db9ec823/compiler_builtins-0.1.49/./lib/builtins/cpu_model.c:786: undefined reference to `getauxval'
          collect2: error: ld returned 1 exit status

  = help: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

error: build failed

Expected outcome

Step 6 should succeed :-)

@tonyg
Copy link
Author

tonyg commented Oct 8, 2021

(For reference, --target=armv7-unknown-linux-musleabihf builds work, even with nightly.)

@tonyg
Copy link
Author

tonyg commented Oct 13, 2021

OK, after rather a lot of flailing about, I've managed to get foo to build with nightly. TL;DR: I added musl's libc.a to the rustc invocation via ... -C link-arg=/usr/local/aarch64-linux-musl/lib/libc.a. I did this manually by looking at the -vv output of cross and running adjusted docker commands interactively; now I need to figure out how to do it automatically!


Here's a (whitespace-adjusted reformatted-to-multiline-with-backslashes) version of the command that cargo is running inside the docker container:

PATH=$PATH:/rust/bin \
CARGO=/rust/bin/cargo \
CARGO_BIN_NAME=foo \
CARGO_CRATE_NAME=foo \
CARGO_MANIFEST_DIR=/project \
CARGO_PKG_AUTHORS='' \
CARGO_PKG_DESCRIPTION='' \
CARGO_PKG_HOMEPAGE='' \
CARGO_PKG_LICENSE='' \
CARGO_PKG_LICENSE_FILE='' \
CARGO_PKG_NAME=foo \
CARGO_PKG_REPOSITORY='' \
CARGO_PKG_VERSION=0.1.0 \
CARGO_PKG_VERSION_MAJOR=0 \
CARGO_PKG_VERSION_MINOR=1 \
CARGO_PKG_VERSION_PATCH=0 \
CARGO_PKG_VERSION_PRE='' \
CARGO_PRIMARY_PACKAGE=1 \
LD_LIBRARY_PATH='/target/debug/deps:/rust/lib' \
rustc \
--crate-name foo \
--edition=2018 \
src/main.rs \
--error-format=json \
--json=diagnostic-rendered-ansi \
--crate-type bin \
--emit=dep-info,link \
-C embed-bitcode=no \
-C debuginfo=2 \
-C metadata=cba3bae5335ad02d \
-C extra-filename=-cba3bae5335ad02d \
--out-dir /target/aarch64-unknown-linux-musl/debug/deps \
--target aarch64-unknown-linux-musl \
-C linker=aarch64-linux-musl-gcc \
-C incremental=/target/aarch64-unknown-linux-musl/debug/incremental \
-L dependency=/target/aarch64-unknown-linux-musl/debug/deps \
-L dependency=/target/debug/deps \
-C link-arg=-lgcc

To this I added the following, at the end:

... \
-C link-arg=/usr/local/aarch64-linux-musl/lib/libc.a

So, the whole docker invocation that successfully built me an executable nightly foo was:

"/usr/bin/docker" "run" \
    "--userns" "host" \
    "-e" "LDFLAGS" \
    "-e" "PKG_CONFIG_ALLOW_CROSS=1" \
    "--rm" \
    "--user" "1000:1000" \
    "-e" "XARGO_HOME=/xargo" \
    "-e" "CARGO_HOME=/cargo" \
    "-e" "CARGO_TARGET_DIR=/target" \
    "-e" "USER=tonyg" \
    "-e" "CROSS_RUNNER=" \
    "-v" "/home/tonyg/.xargo:/xargo:Z" \
    "-v" "/home/tonyg/.cargo:/cargo:Z" \
    "-v" "/cargo/bin" \
    "-v" "/home/tonyg/nobackup/build/issue-598/foo:/project:Z" \
    "-v" "/home/tonyg/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu:/rust:Z,ro" \
    "-v" "/home/tonyg/nobackup/build/issue-598/foo/target:/target:Z" \
    "-w" "/project" \
    "-i" "rustembedded/cross:aarch64-unknown-linux-musl-0.2.1" \
    "sh" \
    "-c" \
    "PATH=$PATH:/rust/bin CARGO=/rust/bin/cargo CARGO_BIN_NAME=foo CARGO_CRATE_NAME=foo CARGO_MANIFEST_DIR=/project CARGO_PKG_AUTHORS='' CARGO_PKG_DESCRIPTION='' CARGO_PKG_HOMEPAGE='' CARGO_PKG_LICENSE='' CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=foo CARGO_PKG_REPOSITORY='' CARGO_PKG_VERSION=0.1.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=0 CARGO_PKG_VERSION_PRE='' CARGO_PRIMARY_PACKAGE=1 LD_LIBRARY_PATH='/target/debug/deps:/rust/lib' rustc --crate-name foo --edition=2018 src/main.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type bin --emit=dep-info,link -C embed-bitcode=no -C debuginfo=2 -C metadata=cba3bae5335ad02d -C extra-filename=-cba3bae5335ad02d --out-dir /target/aarch64-unknown-linux-musl/debug/deps --target aarch64-unknown-linux-musl -C linker=aarch64-linux-musl-gcc -C incremental=/target/aarch64-unknown-linux-musl/debug/incremental -L dependency=/target/aarch64-unknown-linux-musl/debug/deps -L dependency=/target/debug/deps -C link-arg=-lgcc -C link-arg=/usr/local/aarch64-linux-musl/lib/libc.a"

(The last long line there is just a one-line version of the rustc command just above.)

So, looking forward:

  1. Maybe there could be a workaround involving somehow arranging for -C link-arg=/usr/local/aarch64-linux-musl/lib/libc.a to be given to rustc through the whole cross-to-docker-to-cargo-to-rustc chain?
  2. What changed recently to cause this problem? It's not clear to me whether cross or cargo or something else is at fault.

@tonyg
Copy link
Author

tonyg commented Oct 13, 2021

(Also: for debugging this I have found commands like cross -vv build -vv ... to be useful, with verbose turned on both for cross and for cargo)

@tonyg
Copy link
Author

tonyg commented Oct 13, 2021

Automated Workaround

In Cross.toml:

[build.env]
passthrough = [
    "RUSTFLAGS",
    "CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS",
]

and then

CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS="-C link-arg=/usr/local/aarch64-linux-musl/lib/libc.a" \
    cross -vv build -vv --target=aarch64-unknown-linux-musl

Interesting points to note:

  1. I didn't put -lgcc in there. Setting RUSTFLAGS seems to override the -lgcc that is put in there by default. Could it be that, for musl builds, -lgcc isn't needed? Certainly for foo it seems to be OK, but maybe more complex programs still need it? Building with RUSTFLAGS="-C link-arg=-lgcc -C link-arg=/usr/local/aarch64-linux-musl/lib/libc.a" also works.

  2. There's both RUSTFLAGS and CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS in Cross.toml. It doesn't work without both, if you want to use CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS! If you're happy using plain RUSTFLAGS, then having just RUSTFLAGS in Cargo.toml works fine.

An alternative workaround

Courtesy of @hkratz here, is to use RUSTFLAGS="-Zgcc-ld=lld". You'll still need the Cross.toml above, but with an alternative RUSTFLAGS setting:

CARGO_TARGET_AARCH64_UNKNOWN_LINUX_MUSL_RUSTFLAGS="-Zgcc-ld=lld" \
    cross build --target=aarch64-unknown-linux-musl

Relevant links

@tonyg
Copy link
Author

tonyg commented Oct 13, 2021

So one possible general (?) fix might be to modify aarch64-linux-musl-gcc.sh to put in the path to libc, the way it already special-cases -lgcc for older toolchains, but is that the right thing to do? Is it even a toolchain version change that caused the issue? There seem like an awful lot of moving pieces, and all I know is that roughly a week ago this started happening overnight. I'm not sure what the correct test for adding in a special-casing of libc would be.

@Emilgardis
Copy link
Member

Related to rust-lang/libc#2450, this is probably an issue in libc and not cross

@Emilgardis
Copy link
Member

Does this issue disappear if you pin libc to say 0.2.102 ?

@tonyg
Copy link
Author

tonyg commented Oct 15, 2021

Does this issue disappear if you pin libc to say 0.2.102 ?

No. If I add a dependency libc = "=0.2.102" to Cargo.toml in a fresh foo project (as above), with nightly in rust-toolchain, I get the same error about undefined reference to getauxval. (I'm skeptical that the libc crate is to blame, but I definitely don't have deep expertise here -- doesn't the fact that this occurs in a fresh project without any dependencies mean that the libc crate isn't even present?)

@Emilgardis
Copy link
Member

(I'm skeptical that the libc crate is to blame, but I definitely don't have deep expertise here -- doesn't the fact that this occurs in a fresh project without any dependencies mean that the libc crate isn't even present?)

You are correct, I missed that the project didn't use libc, just assumed so. My bad!

The issue is now tracked in rust-lang/rust#89626 , closing this issue.

@Alexhuszagh Alexhuszagh added bug upstream A-arm Area: ARM targets labels Nov 6, 2022
xabolcs referenced this issue in xabolcs/mtk_uartboot Mar 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-arm Area: ARM targets bug upstream
Projects
None yet
Development

No branches or pull requests

3 participants