SEGFAULT on MacOS with scikit-learn==1.0.0

[PrimozGodec]

xgboost segfaults in combination with scikit-learn==1.0.0 and works fine with scikit-learn==0.24

Steps to reproduce:

1. Install scikit-learn==1.0.0 and xgboost==1.5.0
2. Run this simple script

    import numpy as np
    import xgboost
    
    x = np.random.random((150, 4))
    y = np.random.randint(0, 3, 150)
    
    learner = xgboost.XGBClassifier()
    learner.fit(x, y)
   

I found out that scikit-learn brings its own version of libomp library together with the package. It is stored in sklearn/.dylibs/libomp.dylib. When I remove it from sklearn segfault does not happen. When libomp is present in sklearn and I remove a global version from the computer xgboost does not work since it cannot find the library. It shows that xgboost is actually not using the libomp library from scikit-learn but there must be some coincidence.

When downgrading scikit-learn to 0.24 it still ships its own version of libomp but the segfaul does not happen.

[trivialfis]

Any chance this is the same issue with #7039 ?

[PrimozGodec]

I am not sure. From what can I read there downgrading limpomp to 11 helped. In my case, it does not.
Also, the mentioned issue was opened before scikit-learn=1.0.0 was released. My issue started to appear after scikit-learn==1.0.0 release (Sep 24). As I mentioned, with an older version of scikit-learn everything works fine.

[trivialfis]

@hcho3 might have better insight. My guess is just scikit-learn < 1.0 ships older libomp while scikit-learn==1.0 ships libomp >= 12.0.

[PrimozGodec]

With both versions of scikit-learn I get the same version for dylib

        /DLC/sklearn/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)

when I call otool -L <path to dylib>.

It also seems that xgboost is not using scikit's version of libomp (when I uninstalled libomp with brew and left scikits libomp xgboost didn't work).

[StrikerRUS]

@trivialfis

My guess is just scikit-learn < 1.0 ships older libomp while scikit-learn==1.0 ships libomp >= 12.0.

You are absolutely right! Please refer to microsoft/LightGBM#4632 (comment) and scikit-learn/scikit-learn#21227.

[trivialfis]

Bug fix in scikit-learn is merged.