LLamaSharp runtime binaries don't support Rosetta2

[abhiaagarwal]

Description

Hi there,

I'm attempting to integrate LLamaSharp with a C#-based game that is stuck with Rosetta emulation on M1 Macs. Unfortunately, the regular builds are compiled with AVX (I believe?) which Rosetta can't emulate, causing a crash. I compiled with minimal support from the Llama.cpp repo and it works fine.

I'm not sure if this is something that needs to be addressed by the library itself, but felt worth noting here.

[martindevans]

There should be non AVX binaries (it's the ones in this folder).

Try running your application with this at the start:

NativeLibraryConfig
   .All
   .WithLogCallback((level, message) =>
   {
       Console.WriteLine($"[llama {level}]: {message.TrimEnd('\n')}");
   });

It will log out what the binary selection process is doing, which should help diagnose why it's choosing AVX over noavx.

[abhiaagarwal]

Those are .so files, right? I don't think MacOS can load those. It's only looking for the dylibs based on

LLamaSharp/LLama/Native/Load/NativeLibraryUtils.cs

Line 23 in c51a65f

// Get platform specific parts of the path (e.g. .so/.dll/.dylib, libName prefix or not)

.

[martindevans]

Aha yes sorry I forgot about that. In that case yes you're right, as you can see here the MacOS+x64 binaries are compiled with AVX support.

If you're interested in submitting a PR to fix this it should be fairly simple.

• Here is an example of adding a new step to the build matrix for noavx.
• Here the files are being copied into their final location, you'll need to add somewhere for the noavx binaries to go (probably add the noavx binaries into that folder, and create an avx folder for the existing ones).
• Runtime loading will need modifying to load the relevant binary. That's probably in here. @AsakusaRinne might have a better idea then me since they worked on this recently.

[abhiaagarwal]

Yeah, I already hacked together a fix in my codebase – the only problem is that it's relatively convoluted to figure out "whether we're running under rosetta emulation without AVX support", because AVX works fine for regular Intel-based Macs (even though the hardware platform is increasingly rare).

[martindevans]

Does this not work under Rosetta?

[AsakusaRinne]

Runtime loading will need modifying to load the relevant binary. That's probably in here. @AsakusaRinne might have a better idea then me since they worked on this recently.

It's good to make changes at this place. It's better to split this class into NativeLibraryFallback and NativeLibraryForMac because Rosetta only appears with MAC. But don't worry, I can do it after your PR. :)

[abhiaagarwal]

Does this not work under Rosetta?

Yep, I didn't think about tackling it at this angle. I'll open a PR later today. Thank you :)