fail when using "pip install POT"

[Adoni]

When I use "pip install POT", it failed. It depended on Cython. However, it seems that it forgets to tell pip that it depends on Cython.

I solve this problem by install Cython first. However, if we write both Cython and POT into requirements.txt, the installation will fail.

Could anyone solve that?

[rflamary]

Hello @Adoni and thank you for the bug report.

This is very surprising since I cannot seem to reproduce the bug. On what platform are you? When using pip, the variable install_requires defined in the setup.py should install automatically cython when building POT.

[Adoni]

It's mac os in my computer and I'm using anaconda.

However, my friend reproduced this bug in Linux, who didn't use Anaconda. So to some degree I think this is a common problem.

[rflamary]

Hello i'm sorry but I don't have enough information to reproduce the bug.

On anaconda we recommend to use conda forge that take care of all compilation.

We now have an issue template with the list of required information :
https:/rflamary/POT/blob/master/.github/ISSUE_TEMPLATE/bug_report.md

[leotrs]

I have been able to reproduce the bug by installing POT from a requirements.txt file. Example bash session follows.

leo@lily: test-pot$ mkvirtualenv test-pot
Running virtualenv with interpreter /usr/bin/python2
New python executable in /home/leo/.virtualenvs/test-pot/bin/python2
Also creating executable in /home/leo/.virtualenvs/test-pot/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
virtualenvwrapper.user_scripts creating /home/leo/.virtualenvs/test-pot/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/leo/.virtualenvs/test-pot/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/leo/.virtualenvs/test-pot/bin/preactivate
virtualenvwrapper.user_scripts creating /home/leo/.virtualenvs/test-pot/bin/postactivate
virtualenvwrapper.user_scripts creating /home/leo/.virtualenvs/test-pot/bin/get_env_details

(test-pot) leo@lily: test-pot$ echo "matplotlib==2.0.2
> numpy==1.13.1
> networkx==1.11
> scikit-learn==0.18.2
> scipy==0.19.1
> pandas==0.19.1
> POT==0.5.1" > requirements.txt

(test-pot) leo@lily: test-pot$ cat requirements.txt 
matplotlib==2.0.2
numpy==1.13.1
networkx==1.11
scikit-learn==0.18.2
scipy==0.19.1
pandas==0.19.1
POT==0.5.1

(test-pot) leo@lily: test-pot$ pip install -r requirements.txt 
<suppressed output...>
Collecting POT==0.5.1 (from -r requirements.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/28/4b/7aaa1f840a359f5953dd378e0237fa8faf9b0a415ff7282b7375fbe68d27/POT-0.5.1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-zvHRKm/POT/setup.py", line 7, in <module>
        from Cython.Build import cythonize
    ImportError: No module named Cython.Build
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-zvHRKm/POT/

[ncourty]

Hi @leotrs, are you working on OSX Mojave ? It seems the problem comes from a bad include directory with the compiler used by Cython. See this issue #71 (comment) with the patch that was supposed to work at that time...

[ncourty]

wait I might have reacted too quickly.... Is cython installed on your side ? Maybe we just forgot to add it in the requirements as it usually ships with most python distrib

[leotrs]

No. I'm on Ubuntu 16.04. I should clarify that I was able to install POT by (i) removing it from the requirements.txt file, (ii) installing the remaining requirements using pip install -r requirements.txt, and (iii) installing POT on its own pip install pot.

[leotrs]

I believe it must be part of the requirements, since doing just pip install pot works for me. The only time it doesn't work is when it is being installed from a requirements.txt file.

[leotrs]

Any updates on this?

[rflamary]

Hello @leotrs,

Sorry for the late reply but we have been quite busy. I have seen the discussion on the bug you refered and I can tell you the POT is actively maintained. e have peaks of activity followed by limited availability but we try to handle bugs in a timely manner (except this one apparently ;) )

This is a weird bug and I am trying to reproduce your bug but i'm on ubuntu 18.04.

This is what i have

[:~/test2] master ± virtualenv test-pot
Using base prefix '/usr'
New python executable in /home/rflamary/test2/test-pot/bin/python3
Also creating executable in /home/rflamary/test2/test-pot/bin/python
Installing setuptools, pip, wheel...done.
[:~/test2] master ± source test-pot/bin/activate
[:~/test2] [test-pot] master ± echo "matplotlib==2.0.2
> > numpy==1.13.1
> > networkx==1.11
> > scikit-learn==0.18.2
> > scipy==0.19.1
> > pandas==0.19.1
> > POT==0.5.1" > requirements.txt
[:~/test2] [test-pot] master ±  cat requirements.txt 
matplotlib==2.0.2
> numpy==1.13.1
> networkx==1.11
> scikit-learn==0.18.2
> scipy==0.19.1
> pandas==0.19.1
> POT==0.5.1
[:~/test2] [test-pot] master ± cat requirements.txt 
matplotlib==2.0.2
numpy==1.13.1
networkx==1.11
scikit-learn==0.18.2
scipy==0.19.1
pandas==0.19.1
POT==0.5.1
[:~/test2] [test-pot] master ±  pip install -r requirements.txt 
Collecting matplotlib==2.0.2 (from -r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/60/d4/6b6d8a7a6bc69a1602ab372f6fc6e88ef88a8a96398a1a25edbac636295b/matplotlib-2.0.2-cp36-cp36m-manylinux1_x86_64.whl
Collecting numpy==1.13.1 (from -r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/59/e2/57c1a6af4ff0ac095dd68b12bf07771813dbf401faf1b97f5fc0cb963647/numpy-1.13.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting networkx==1.11 (from -r requirements.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/d3/2c/e473e54afc9fae58dfa97066ef6709a7e35a1dd1c28c5a3842989322be00/networkx-1.11-py2.py3-none-any.whl
Collecting scikit-learn==0.18.2 (from -r requirements.txt (line 4))
  Using cached https://files.pythonhosted.org/packages/d1/51/ca5be39c576c981cf8b8359fcb1ee49ca43d59b833415205cec5ef5c0fcd/scikit_learn-0.18.2-cp36-cp36m-manylinux1_x86_64.whl
Collecting scipy==0.19.1 (from -r requirements.txt (line 5))
  Using cached https://files.pythonhosted.org/packages/0e/46/da8d7166102d29695330f7c0b912955498542988542c0d2ae3ea0389c68d/scipy-0.19.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting pandas==0.19.1 (from -r requirements.txt (line 6))
Collecting POT==0.5.1 (from -r requirements.txt (line 7))
Collecting cycler>=0.10 (from matplotlib==2.0.2->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/f7/d2/e07d3ebb2bd7af696440ce7e754c59dd546ffe1bbe732c8ab68b9c834e61/cycler-0.10.0-py2.py3-none-any.whl
Collecting pytz (from matplotlib==2.0.2->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl
Collecting python-dateutil (from matplotlib==2.0.2->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting pyparsing!=2.0.0,!=2.0.4,!=2.1.2,!=2.1.6,>=1.5.6 (from matplotlib==2.0.2->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/de/0a/001be530836743d8be6c2d85069f46fecf84ac6c18c7f5fb8125ee11d854/pyparsing-2.3.1-py2.py3-none-any.whl
Collecting six>=1.10 (from matplotlib==2.0.2->-r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting decorator>=3.4.0 (from networkx==1.11->-r requirements.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/f1/cd/7c8240007e9716b14679bc217a1baefa4432aa30394f7e2ec40a52b1a708/decorator-4.3.2-py2.py3-none-any.whl
Collecting cython (from POT==0.5.1->-r requirements.txt (line 7))
  Using cached https://files.pythonhosted.org/packages/e1/fd/711507fa396064bf716493861d6955af45369d2c470548e34af20b79d4d4/Cython-0.29.6-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: six, cycler, pytz, python-dateutil, pyparsing, numpy, matplotlib, decorator, networkx, scikit-learn, scipy, pandas, cython, POT
Successfully installed POT-0.5.1 cycler-0.10.0 cython-0.29.6 decorator-4.3.2 matplotlib-2.0.2 networkx-1.11 numpy-1.13.1 pandas-0.19.1 pyparsing-2.3.1 python-dateutil-2.8.0 pytz-2018.9 scikit-learn-0.18.2 scipy-0.19.1 six-1.12.0

And it automatically installed cython which seemed to be the bug in what you sent us. I tried it with python3.6 and python2.7 and it worked.

I'm not familiar with virtualenvwrapper but i don't this the discrepancy come from here. So it might be a problem of version from pip? The install_requires parameter has only been added recently and if you have an old pip he might discard it and not install cython that is required to compile POT. if you want to be sure to compile whatever the pip version you can always add cython in the requirement (I know nobody wants that but we do not provide pre-compiled eggs on pipy).

Edit: reran the test after cleaning PYTHONPATH variable.

[leotrs]

@rflamary thank you for looking into this. I don't think it's the version of pip since we are replicating the bug across a number of computers.

[rflamary]

OK I managed to reproduce the problem.
This occurs on python2.7 when i force it for the virtual environment

virtualenv --python=/usr/bin/python2.7 test_pot_py27

It works on python 3.6...

I'm wondering if it is a problem related to pypa/pip#2478
Its the same pip version for both, this is a weird bug. install_requires should work on both 2.7 and 3.6 but it seems to fail installing Cython before POT on 2.7...

edit: wrong link

[joshua-gould]

Here's a Dockerfile that shows the package failing using pip install with the error File "/tmp/pip-build-y26thcux/POT/setup.py", line 7, in
from Cython.Build import cythonize
ModuleNotFoundError: No module named 'Cython'

FROM ubuntu:latest
RUN apt-get -qq update && apt-get -qq -y install python3.7 python3-pip build-essential
RUN pip3 install POT

[rflamary]

This is perfect for debug thank you.

I will have a look (probably after the NeurIPS deadline) but i think it might be corrected by properly defining build dependencies as discussed https://packaging.python.org/specifications/declaring-build-dependencies/ and https://www.python.org/dev/peps/pep-0518/.

[rflamary]

Hello,

This problem is a known bug from pypi discussed here
pypa/pip#2478

and here
pypa/setuptools#391

Basically in order to install POT you need to have cython and numpy already installed because they are dependency at build time. From what i saw this is a common problem. I think it could be addressed if we provided compiled eggs (since they won't require building anymore) but its a lot of work.

a working Dockerfile below

FROM ubuntu:latest
RUN apt-get -qq update && apt-get -qq -y install python3.7 python3-pip build-essential git
RUN pip3 install cython numpy
RUN pip3 install POT

I agree it's not optimal but i don't think we can do anything until the question is handled by pypi