Changed behavior with --extra-index-url with pip 9

[hjc]

• Pip version: 9.0.1
• Python version: 3.5.2+
• Operating System: Debian Stretch (testing)

Description:

We have a private internal package we use for our projects that is hosted on our own internal server. That packaging server only has our private packages and is not a full PyPI mirror, because it's not what we need. In the past, I have been able to run:

pip install -r requirements.txt --extra-index-url http://pip.croscon.com:8088/simple/ --trusted-host pip.croscon.com

and it works fine. However, now when I run this command, I get an error from pip saying it can't find a specific package on pip.croscon.com. This package is not our internal package and is one we normally pull from PyPI. This critically errors and fails. Going back to pip 8 fixed this.

What I've run:

(.env) hayden@beard-machine ~/g/zvm-demo> pip --version
pip 9.0.1 from /home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages (python 3.5)
(.env) hayden@beard-machine ~/g/zvm-demo> pip install -r requirements.txt  \
                                            --extra-index-url http://pip.croscon.com:8088/simple/  \
                                            --trusted-host pip.croscon.com
Collecting Flask-Celery-Helper==1.1.0 (from -r requirements.txt (line 1))
Exception:
Traceback (most recent call last):
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/commands/install.py", line 353, in run
    wb.build(autobuilding=True)
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/req/req_set.py", line 554, in _prepare_file
    require_hashes
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/req/req_install.py", line 278, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/index.py", line 465, in find_requirement
    all_candidates = self.find_all_candidates(req.name)
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/index.py", line 423, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/index.py", line 568, in _get_pages
    page = self._get_page(location)
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/index.py", line 683, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/home/hayden/gits/zvm-demo/.env/lib/python3.5/site-packages/pip/index.py", line 795, in get_page
    resp.raise_for_status()
  File "/home/hayden/gits/zvm-demo/.env/share/python-wheels/requests-2.11.1-py2.py3-none-any.whl/requests/models.py", line 862, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
pip._vendor.requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://pip.croscon.com:8088/simple/flask-celery-helper/
# now swap to pip 8
(.env) hayden@beard-machine ~/g/zvm-demo> pip install -U "pip>=8.0.0,<9.0.0"
Collecting pip<9.0.0,>=8.0.0
  Downloading pip-8.1.2-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 895kB/s 
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
Successfully installed pip-8.1.2
(.env) hayden@beard-machine ~/g/zvm-demo> pip install -r requirements.txt  \
                                            --extra-index-url http://pip.croscon.com:8088/simple/  \
                                            --trusted-host pip.croscon.com
  ... omitted for brevity; but it was pip working as expected ...
Successfully installed Flask-0.12 Flask-Celery-Helper-1.1.0 Flask-Classful-0.13.1 Flask-Login-0.4.0 Flask-Marshmallow-0.7.0 Jinja2-2.8 MarkupSafe-0.23 Werkzeug-0.11.11 amqp-1.4.9 anyjson-0.3.3 arrow-0.10.0 billiard-3.3.0.23 blinker-1.4 celery-3.1.25 click-6.6 eventlet-0.19.0 fleaker-0.1.4 google-auth-0.5.0 google-auth-httplib2-0.0.2 google-cloud-core-0.22.1 google-cloud-storage-0.22.0 googleapis-common-protos-1.5.0 greenlet-0.4.11 gunicorn-19.6.0 httplib2-0.9.2 invoke-0.12.1 itsdangerous-0.24 kombu-3.0.37 marshmallow-2.10.5 marshmallow-jsonschema-0.3.0 oauth2client-4.0.0 pendulum-0.7.0 protobuf-3.1.0.post1 pyasn1-0.1.9 pyasn1-modules-0.0.8 python-dateutil-2.6.0 pytz-2016.10 pytzdata-2016.10 redis-2.10.5 rsa-3.4.2 six-1.10.0 tzlocal-1.3
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

[hjc]

Hello?

Any response? Hit this again today which reminded me of the problem at hand.

[hjc]

Interestingly enough, this is an issue with virtualenv. The pip it installs is just broken by default. Downgrading to pip 8 and then upgrading back to pip 9 fixes it.

Virtualenv version is: 15.1.0.

[teej-minted]

Just hit this exact same issue (private repo, pip install fails saying it cant find the package). We followed the exact same procedure (downgrade to ~8, upgrade back to 9) and it worked. We are all shaking our heads at this.

[dstufft]

Are you using virtualenv that came with Debian? They patch the pip they install with that and the upgrade/downgrade dance ends up forcing you to use the unpatched version from PyPI. It could be an issue with the patching done by Debian.

[arthurdarcet]

This is definitely a bug in the debian patch for pip ; it was reported here https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=837764 a long time ago but went dead. It is still happening today.

[cboylan]

I don't actually think this is completely a downstream bug. The issue is related to how debian unvendors vendored libs, but we trip over that because the exception is pip._vendor.requests.exceptions.HTTPError not pip._vendor.requests.HTTPError.

I'll get a PR up shortly that seems to fix this for debian (and also happens to be more like the other exception handling in index.py).

[cjerdonek]

Reopening as this was found in #6367 to be an issue with pip’s DEBUNDLED section. (Thanks, @cboylan, for your contribution here!)

[cjerdonek]

Addressed by PR #6373 / 5af0bc3.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.