Avoid duplicated loading native library

[frankfliu]

Description

Brief description of what this PR is about

• If this change is a backward incompatible change, why must this change be made?
• Interesting edge cases to note here

[humcqc]

Hello, thanks for the fix,
Could you tell me if this will fix the langchain4j/langchain4j#776

I'm not sure as the exception is throw in the System.load call:
java.lang.UnsatisfiedLinkError: Native Library /Users/gkwan/.djl.ai/tokenizers/0.15.0-0.26.0-osx-aarch64/libtokenizers.dylib already loaded in another classloader
[INFO] at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:201)
[INFO] at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:174)
[INFO] at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2389)
[INFO] at java.base/java.lang.Runtime.load0(Runtime.java:755)
[INFO] at java.base/java.lang.System.load(System.java:1953)
[INFO] at ai.djl.huggingface.tokenizers.jni.LibUtils.loadLibrary(LibUtils.java:76)
[INFO] at ai.djl.huggingface.tokenizers.jni.LibUtils.(LibUtils.java:41)

Could we add a check to see if the library is already loaded or just catch the exception in case of and log a warning ?

Thanks

[frankfliu]

@humcqc

I don't think this will fix your issue. Your issue is a typical java ClassLoader issue. see similar discussion: #179

You are loading the DJL tokenizer into two different ClassLoader, which is not allowed by JVM. You need to make sure the .so file is loaded in the common parent class loader to avoid this bug.

Here is something you can do:

1. Use Thread.currentThread().setContextClassLoader(parentClassLoader), to set a common class loader before you load tokenizer in your thread.
2. or you can write a "NativeHelper" class, and put this class in system classpath, and set system property:

System.setProperty("ai.djl.huggingface.native_helper", "com.examples.NativeHelper");

[humcqc]

thanks @frankfliu , do you think we can apply your first suggestion directly to your LibUtils to find a centralized solution ?