ndarray conversion fails after updating jep, scipy, numpy

[jovrstra]

Hello,

I recently needed to update some packages (pdfo and then numpy and scipy) and also updated jep to version 4.2.0 (not sure what the previous version was, but it was at least a year old. Since then, I cannot seem to import ndarrays to java.
The line in Java that used to work was:
NDArray<double[]> resultVector = (NDArray<double[]>) jep.getValue("get_result_x()");
But now this throws a
Exception in thread "main" java.lang.ClassCastException: class [D cannot be cast to class jep.NDArray ([D is in module java.base of loader 'bootstrap'; jep.NDArray is in unnamed module of loader 'app') at ds.python.PythonSciPy.test(PythonSciPy.java:119) ...

The above get_result_x is a getter applied on a SciPy OptimimizeResult to return the x:
https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.OptimizeResult.html#scipy.optimize.OptimizeResult
Jep is a new SharedInterpreter().

No changes were made to the Java code, I just updated the above packages and changed the referenced jar file to use jep-4.2.0.jar. It worked just prior to these updates.

Can anyone help?

[jovrstra]

Could it be that Jep 4.2.0 was not built with numpy support? How to get a numpy supported version?

[bsteffensmeier]

Could it be that Jep 4.2.0 was not built with numpy support? How to get a numpy supported version?

That is likely the cause. Jep will automatically include numpy support if numpy is present when you build jep so you need to install numpy before jep. You can check if numpy support is enabled by doing something like interpreter.exec("print(jep.JEP_NUMPY_ENABLED)");. If you are using pip to install jep and you want to rebuild after installing numpy you may need to use the --no-cache-dir option for pip or it will reuse your existing jep build without numpy support.

[jovrstra]

Numpy support is not enabled (the above command prints 0), so that is the issue.
I have a problem getting it to work though. I removed Jep (pip uninstall jep), cleared the cache (py -m pip cache purge), double checked that there is no jep folder in my pyhton folder, checked that numpy is installed (Numpy 2.1.1) downloaded the jep-sources, and ran (pip install . --no-cache-dir).
Building seems to work:

Processing c:\users\...\jep-4.2.0.tar\jep-4.2.0\jep-4.2.0
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: jep
  Building wheel for jep (pyproject.toml) ... done
  Created wheel for jep: filename=jep-4.2.0-cp310-cp310-win_amd64.whl size=357425 sha256=8f52d839e3a43c1dfc22036cc591f6c571ba384f864dbc8fc1dd15b45424b5f8
  Stored in directory: C:\Users\...\AppData\Local\Temp\pip-ephem-wheel-cache-enes7qra\wheels\92\02\87\e7b17261aa61b4cb677adab9e7b58990f7f49a1d764657e07c
Successfully built jep
Installing collected packages: jep
Successfully installed jep-4.2.0

But my Java program still seems to show that I'm missing numpy support in jep.
I've updated the Java project with the correct jar file for jep. I have cleaned the project and restarted Eclipse, but it stays the same.
How to build this? (Windows 10 x64)

Edit: uninstalled everything, reinstalled Python 3.12.6, numpy 2.1.1 and tried building Jep. Building works, still no numpy support... I'm baffled...

Edit: building with the --verbose option shows that numpy is not found... everything tells me it is there... what am I missing?

[jovrstra]

How to get Jep to find numpy?

PS C:\Users\...\Downloads\jep-4.2.0.tar> pip install numpy --verbose Using pip 24.2 from C:\Program Files\Python312\Lib\site-packages\pip (python 3.12) Link requires a different Python (3.12.6 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/3a/be/650f9c091ef71cb01d735775d554e068752d3ff63d7943b26316dc401749/numpy-1.21.2.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11) Link requires a different Python (3.12.6 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/5f/d6/ad58ded26556eaeaa8c971e08b6466f17c4ac4d786cd3d800e26ce59cc01/numpy-1.21.3.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11) Link requires a different Python (3.12.6 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/fb/48/b0708ebd7718a8933f0d3937513ef8ef2f4f04529f1f66ca86d873043921/numpy-1.21.4.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11) Link requires a different Python (3.12.6 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/c2/a8/a924a09492bdfee8c2ec3094d0a13f2799800b4fdc9c890738aeeb12c72e/numpy-1.21.5.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11) Link requires a different Python (3.12.6 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/45/b7/de7b8e67f2232c26af57c205aaad29fe17754f793404f59c8a730c7a191a/numpy-1.21.6.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11) Collecting numpy Obtaining dependency information for numpy from https://files.pythonhosted.org/packages/b7/98/5640a09daa3abf0caeaefa6e7bf0d10c0aa28a77c84e507d6a716e0e23df/numpy-2.1.1-cp312-cp312-win_amd64.whl.metadata Downloading numpy-2.1.1-cp312-cp312-win_amd64.whl.metadata (59 kB) Downloading numpy-2.1.1-cp312-cp312-win_amd64.whl (12.6 MB) ---------------------------------------- 12.6/12.6 MB 39.3 MB/s eta 0:00:00 Installing collected packages: numpy Successfully installed numpy-2.1.1

And then:
PS C:\Users\ ...\Downloads\jep-4.2.0.tar\jep-4.2.0> pip install . --no-cache-dir --verbose Using pip 24.2 from C:\Program Files\Python312\Lib\site-packages\pip (python 3.12) Processing c:\users\ ...\downloads\jep-4.2.0.tar\jep-4.2.0 Running command pip subprocess to install build dependencies Using pip 24.2 from C:\Program Files\Python312\Lib\site-packages\pip (python 3.12) Collecting setuptools>=40.8.0 Obtaining dependency information for setuptools>=40.8.0 from https://files.pythonhosted.org/packages/8e/69/f90971743a7cc451b93be8f5eb60b7e506896b7e81e6be6272858b2a9d46/setuptools-75.0.0-py3-none-any.whl.metadata Downloading setuptools-75.0.0-py3-none-any.whl.metadata (6.9 kB) Downloading setuptools-75.0.0-py3-none-any.whl (1.2 MB) ---------------------------------------- 1.2/1.2 MB 12.6 MB/s eta 0:00:00 Installing collected packages: setuptools Successfully installed setuptools-75.0.0 Installing build dependencies ... done Running command Getting requirements to build wheel C:\Users\ ...\Downloads\jep-4.2.0.tar\jep-4.2.0\commands\scripts.py:57: SyntaxWarning: invalid escape sequence '\#' """Copy each script listed in 'self.scripts'; if it's marked as a numpy not found, running without numpy support running egg_info writing src/main/python\jep.egg-info\PKG-INFO writing dependency_links to src/main/python\jep.egg-info\dependency_links.txt writing top-level names to src/main/python\jep.egg-info\top_level.txt reading manifest file 'src/main/python\jep.egg-info\SOURCES.txt' reading manifest template 'MANIFEST.in' adding license file 'LICENSE' adding license file 'AUTHORS' writing manifest file 'src/main/python\jep.egg-info\SOURCES.txt' Getting requirements to build wheel ... done Running command Preparing metadata (pyproject.toml) numpy not found, running without numpy support running dist_info creating C:\Users\ ...\AppData\Local\Temp\pip-modern-metadata-pzdc6gii\jep.egg-info writing C:\Users\ ...\AppData\Local\Temp\pip-modern-metadata-pzdc6gii\jep.egg-info\PKG-INFO writing dependency_links to C:\Users\ ...\AppData\Local\Temp\pip-modern-metadata-pzdc6gii\jep.egg-info\dependency_links.txt writing top-level names to C:\Users\ ...\AppData\Local\Temp\pip-modern-metadata-pzdc6gii\jep.egg-info\top_level.txt writing manifest file 'C:\Users\ ...\AppData\Local\Temp\pip-modern-metadata-pzdc6gii\jep.egg-info\SOURCES.txt' reading manifest file 'C:\Users\ ...\AppData\Local\Temp\pip-modern-metadata-pzdc6gii\jep.egg-info\SOURCES.txt' reading manifest template 'MANIFEST.in' adding license file 'LICENSE' adding license file 'AUTHORS' writing manifest file 'C:\Users\ ...\AppData\Local\Temp\pip-modern-metadata-pzdc6gii\jep.egg-info\SOURCES.txt' creating 'C:\Users\ ...\AppData\Local\Temp\pip-modern-metadata-pzdc6gii\jep-4.2.0.dist-info' Preparing metadata (pyproject.toml) ... done Building wheels for collected packages: jep Running command Building wheel for jep (pyproject.toml) numpy not found, running without numpy support
What is missing that numpy is not found? Is it some path setting? I've tried rebooting before building jep...

Numpy config shows the paths:
numpy-config --cflags: -IC:\Program Files\Python312\Lib\site-packages\numpy\_core\include
and
numpy-config --pkgconfigdir: C:\Program Files\Python312\Lib\site-packages\numpy\_core\lib\pkgconfig

The include files are not in numpy\core\include, but in numpy_core\include\numpy.

I tried with a lower version of numpy (1.26.4), which does put the include files in numpy\core\include, but jep is still not finding numpy... I'm at a complete loss...

[jovrstra]

I managed to build using numpy 1.26.4, and using " python setup.py build" and "python setup.py install". For some reason, when ran from pip, the test for the installation of numpy fails.
I don't know if it would work with numpy 2.1.1, as the include files are not in core\include but in _core\include\numpy (I have the impression that the path in the setup script looks for them in core\include.

[jovrstra]

I close this issue as it is resolved for me.
However, there still seems to be an issue installing using pip, and potentially with numpy 2.1.1.

[jovrstra]

Reopening the issue as installing via python rather than pip will be deprecated, so the issue of building with numpy will be an issue.
DEPRECATION: Loading egg at c:\program files\python312\lib\site-packages\jep-4.2.0-py3.12-win-amd64.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at pypa/pip#12330

[ndjensen]

Thanks for sharing what you found. We had not tried building Jep with numpy 2 yet. @bsteffensmeier opened pull request #562 to get the build to properly detect numpy 2.

[jovrstra]

No problem. I hope it helps. Weird thing is that with numpy 1.26.4 (with the include files in the expected locations), pip install . fails to detect numpy, but python setup.py build works.
(I think we can now safely close this thread :-))