NDK link-time optimization fails missing llvmgold.so, but contains llvmgold.dylib

[JakeWharton]

I'm attempting to add -flto for link-time optimization to my cFlags and ldFlags (using the new Gradle experimental plugin) but receiving a failure to load LLVMgold.so at link time:

/Users/jw/android-sdk/ndk-bundle/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/lib/gcc/aarch64-linux-android/4.9.x/../../../../aarch64-linux-android/bin/ld: /Users/jw/android-sdk/ndk-bundle/toolchains/llvm/prebuilt/darwin-x86_64/bin/../lib64/LLVMgold.so: error loading plugin: dlopen(/Users/jw/android-sdk/ndk-bundle/toolchains/llvm/prebuilt/darwin-x86_64/bin/../lib64/LLVMgold.so, 2): image not found

A quick look into the NDK reveals that the lib64/ folder contains an LLVMgold.dylib file, but not the .so:

$ find ~/android-sdk/ndk-bundle -name 'LLVMgold*'
/Users/jw/android-sdk/ndk-bundle/toolchains/llvm/prebuilt/darwin-x86_64/lib64/LLVMgold.dylib

Is the .so missing, or is there a different way to enable LTO that uses this dynamic lib instead?

[JakeWharton]

This is using clang 3.8.

[DanAlbert]

@stephenhines do we just configure something assuming everything is Linux when we build clang, or does upstream clang actually have the wrong? Maybe it's trying .so because the target is Linux?

[stephenhines]

We do only target baseline Linux for our configuration (multiple layers of config.h), so perhaps this is a misconfiguration issue. I am not sure whether we will update the existing configuration logic now that we are switching to a more upstream-like build process.

[stephenhines]

https://android-review.googlesource.com/233868 should be the fix for this. In the meantime, you might be able to workaround this by symlinking to a .so filename (i.e. LLVMGold.so -> LLVMGold.dylib in that directory).

[JakeWharton]

Thanks!

That got me slightly farther along, but the linker now fails with an inability to recognize the format of the .o:

/Users/jw/dev/square/duktape-android/duktape/build/intermediates/objectFiles/arm64-v8aReleaseSharedLibrary/duktapeMainC/72j2t6goef75gvdzb76w4b6so/duktape.o: file not recognized: File format not recognized
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

If I had to guess it's because libLTO.so isn't included in the NDK? I'm not sure if that's something that usually comes standard with a native toolchain like this or not (vastly inexperienced with building native libraries).

[kneth]

I have the same issue. Or to be more specific:

• With ARM64, I get the same error message
• With ARM v7a, I get "file format not recognized" for the .o files
• With x86, I can compile and link, but the produced .so file does not work as the dynamic linker on my emulated device cannot find any methods

[suikki]

On Windows if I specify -flto when using clang I get:

LLVMgold.so: could not load plugin library: unable to load dll

[DanAlbert]

The Windows issue is also covered by the fix @stephenhines posted. Given the "file format not recognized" issues though, that probably isn't sufficient.

[thanm]

@JakeWharton, @kneth -- are you 100% sure that the linker being used for your build is ld.gold and not ld.bfd? Looking at the default for prebuilts on the platform it looks like "ld" defaults to gold for linux but bfd for darwin/aarch64.

[kneth]

I have tried with R12 today - using Linux (well, Ubuntu 14.04 under vagrant/OS X). I am able to compile and link but something is wrong with the produced .so files. The dynamic linker cannot find any symbols.

[DoDoENT]

After creating symlinks as @stephenhines suggested, I now get following linker errors:

/Users/dodo/android-sdks/android-ndk-r12/toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/lib/gcc/aarch64-linux-android/4.9.x/../../../../aarch64-linux-android/bin/ld.gold: error: ./obj/local/arm64-v8a/libMyStaticLib.a: no archive symbol table (run ranlib)

for every static library in my project and

/var/folders/fc/38fp3j9x3dd74ng4fl8jhjtr0000gn/T/lto-llvm-2b6578.o:ld-temp.o:function myFunction: error: undefined reference to 'mySymbol'

for every single symbol in my entire code.

When disabling LTO or when using GCC, project compiles without errors.

Something is obviously wrong with clang.

[DanAlbert]

I've seen this happen before when ndk-build wrongly uses the system's ar (specifically on Darwin because it's BSD ar instead of GNU ar) instead of the one we ship in the NDK. Can you check that the correct ar was invoked for creating libMyStaticLib.a? I've certainly fixed bugs in ndk-build for that in the past.

[DoDoENT]

I can confirm that toolchains/aarch64-linux-android-4.9/prebuilt/darwin-x86_64/bin/aarch64-linux-android-ar was used for creating all my static libs (checked with ndk-build -n).

This happens with both r12, r12b and r11c. When GCC is used, LTO works.

[DoDoENT]

This confuses me a lot because 14 days ago I reported issue #137 and after suggestions from @DanAlbert, I've managed to enable LTO for clang on one of my projects (without symlinks suggested by @stephenhines). This means that either then I had a dirty build (actually built with gcc and then forgot to clean after changing compiler) or I was using NDK older than r11c (which I doubt).

Can anyone else reproduce this issue on MacOS X with their project?

[DoDoENT]

I reproduced the same error on ArchLinux using NDK r12b, so this is not MacOSX-specific error (I mean no archive symbol table linker errors).

[kneth]

@DoDoENT I don't recall the actual error message but it sounds like you have reach the same conclusion as me :-(

[vvaldik]

I have "no archive symbol table" error too
The strange thing is than toolchain is 'clang' and cpp's are compiled with clang, but when it comes to link, error shows:
.../android-ndk-r12b/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin/ld.gold: error: ./obj/local/armeabi-v7a/libui.a: no archive symbol table (run ranlib)

So looks like linker is taken from gcc 4.9 toolchain, but my toolchain is 'clang'

[ahyangyi]

@DoDoENT @kneth @vvaldik
Using -fuse-ld=gold and refraining from using ranlib by defining it to /bin/true should eliminate that problem.

[DoDoENT]

This issue is also present in r13.

@ahyangyi, setting CMAKE_RANLIB to /usr/bin/true does not help. I still get undefined reference to every single symbol in my code, as already mentioned in my previous comment. I am using cmake bundled with latest Android Studio (v3.6.3155560) and cmake toolchain bundled with NDK r13.

[DoDoENT]

I've also tried configuring cmake to use GCC instead of clang and then I get following output for each compilation unit:

BFD: CMakeFiles/path/to/source-file.o: plugin needed to handle lto object

And linker fails with undefined reference to each and every symbol in the code. BTW, I double checked that linker command line contains -fuse-ld=gold, yet above mentioned error indicates BFD is used when packing object files in static libraries. Any idea why that happens? Maybe it is also related to clang problem.

[DanAlbert]

Our basic sample builds with r13b (might have worked earlier, don't have r12 on this machine atm), albeit not out of the box.

It looks like the only step necessary (that isn't also necessary on Linux) is that I had to ln -s LLVMgold.dylib LLVMgold.so in $NDK/toolchains/llvm/prebuilt/darwin-x86_64/lib64.

The remaining issues are caused by arm64 still defaulting to ld.bfd, and gold not being supported at all for mips/mips64. Example from #242 (comment)

# Gold does not support mips or mips64, but gold is needed for LTO with Clang.
ifeq (,$(filter $(TARGET_ARCH_ABI),mips mips64)
    LOCAL_CFLAGS += -flto
    LOCAL_LDFLAGS := -flto -fuse-ld=gold
endif

You'll have to ask someone else how to write that for gradle :)

@stephenhines: can you get the dlopen for LLVMgold.so fixed for darwin (and presumably Windows too)? It seems that's the only thing blocking this.

[stephenhines]

This has no chance of making r14, because we are having trouble stabilizing even the latest Clang currently for the platform, and we won't deploy again until that happens. I also still don't understand the actual problem here, so someone who has a better clue on Windows (and a working Windows environment) might have a better chance of resolving this.

[stephenhines]

https:/llvm-mirror/clang/blob/master/lib/Driver/Tools.cpp#L2155 is the problem, thanks to some help from Dan. We can fix this in upstream, but he might be able to just alter the filenames to get a quicker fix out.

[DanAlbert]

QA started working on beta 1 this morning, so unfortunately we just missed that, but I'll have a hack (seems we can just ship with as LLVMgold.so and it'll be technically wrong but work fine) for this in beta 2.

[DoDoENT]

Well, just making symlink does not work for projects that have static libs. Here is the minimum example project which demonstrates the issue @kneth, @vvaldik and I have.

So, when LTO is enabled and gold is used, I get following error:

/Users/dodo/android-sdks/ndk-bundle/toolchains/x86_64-4.9/prebuilt/darwin-x86_64/lib/gcc/x86_64-linux-android/4.9.x/../../../../x86_64-linux-android/bin/ld.gold: error: libstring-provider.a: no archive symbol table (run ranlib)
/Users/dodo/Desktop/MyApplication/app/src/main/cpp/native-lib.cpp:12: error: undefined reference to 'StringProvider::provide_string()'

The example is minimal modification of JNI hello project which is created by Android Studio when creating new project and enabling C++ support.

[DanAlbert]

Ah, that's right. Sorry, there's quite a lot in this thread and I missed that part in my re-read. I'll try to find some time to poke at that today.

Some gradle/NDK mismatch issues came up in testing, so I may end up getting the symlinks into beta 1 anyway. If I can find a workaround for the static library issue in time I'll do the same for that.

[DanAlbert]

Okay, the static library issue reproduces even on Linux. I think I should have time to look at this today.

[DanAlbert]

Expanded our test case to catch this: https://android-review.googlesource.com/307477

[DoDoENT]

Any progress on this?

[DanAlbert]

Yes. It seems that we need to use the LLVMgold.so plugin for ar too: http://llvm.org/docs/GoldPlugin.html

I haven't tested this in the NDK yet, but it looks straightforward.

[DanAlbert]

Fix is up for review: https://android-review.googlesource.com/c/318721/

I'll cherry-pick into r14 for beta 2 once that gets approved.

[DoDoENT]

@DanAlbert, this change only refers to ndk-build. What about CMake toolchain? Will it work there too?

[DanAlbert]

I was thinking that a non-android cmake project would need to modify it's CMakeLists.txt file to make it work anyway, and in general our goal is to just match a typical CMake use case, but it seems that in a non-Android case the gold plugin actually gets installed to a location that ar will automatically look in and load from, so I think we can actually do better than the fix I made yesterday. Working on something now.

[DanAlbert]

Okay, better implementation that is build-system agnostic: https://android-review.googlesource.com/c/319310/

[DanAlbert]

(plus a cmake test)

[DanAlbert]

Alright, fixes are in r14.