Pip install from internal pypi server containing git links fails with pypi 19.1

[K377U]

Environment

• pip version: 19.1
• Python version: Python 3.6.7
• OS: Ubuntu 18.04

Description
We have pypi server that contains links to git:

git+ssh://<url>[email protected]#egg=<package>-0.1.0.201810251140

pip install <package> fails with

  GitLab: The project you were looking for could not be found.
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.

Expected behavior
pip install <package> has been working and does work if I install pip<19.1

How to Reproduce

To reproduce this one needs local pypi server.

Add git+ssh://<url>[email protected]#egg=<package>-0.1.0.201810251140 style link to pypi server where that version number is a git tag.

Then:

virtualenv --python python3 env
. env/bin/activate
pip install 'pip==19.1'
pip install -vvv <package>

Output:

git clone -q ssh://<url>.git%400.1.0.201810251140 <temp>

Running same with pip<19.1 outputs

git clone -q ssh://<local_path>.git <temp>

It looks like the problem is that %400.1.0.201810251140 left in the git link.

[cjerdonek]

This looks to be related to this issue: #6245

It looks like @ should be passed as a safe character to urllib_parse.quote().

@nicolasbock Do you have time to take a look?

[K377U]

I did some debug and end up this location:

index.py

This is the place where @ is changed to %40. It might be that this part should be parsed before here?

[cjerdonek]

urllib.parse.quote() supports passing "safe" characters that shouldn't be quoted: https://docs.python.org/3/library/urllib.parse.html#urllib.parse.quote
So @ can be added to that I think.

[cjerdonek]

Alternatively, the URL should be unquoted prior to parsing. (I haven't really looked at the code yet, so I'm just suggesting possibilities.)

[K377U]

@cjerdonek I tried this

path = urllib_parse.quote(urllib_parse.unquote(result.path), safe="/@")

and it fixes the problem.

[cwharland]

We are hitting this as well. In our case the private pypi is backed by AWS s3 and uses pre-signed s3 URLs as download links. for pip < 19.1 this works for us while for 19.1 we get 403. We can curl the links just fine.

I don't believe our URLs are quoted upon return from our private pypi service that successfully allows pip to find all the versions. I'm a bit stumped as to the new valid format of the download URL.

[nicolasbock]

@cjerdonek Maybe something like #6440 ?

[cwharland]

It seems more related to / perhaps. We're seeing a message like:

ERROR: HTTP error 403 while getting FAILING_URL ( from VALID_URL )

Where the difference is that FAILING_URL has / in the access keys/signed creds and VALID_URL has them correctly as %2F. So the quote change somehow makes it such that the token parts of the URL are now improperly formatted.

our URLs are of the form https://domain_name.org/TOKEN/package_name/package-version.tar.gz

and in the error message it looks as if TOKEN is unquoted.

Update:
Ah I see in #6440 that both / and @ are protected. In that case I'm wondering if the valid auth token character of / will then break the subsequent download.

[nicolasbock]

Do you mean to say that the token has a quoted / which becomes unquoted?

[cwharland]

Yes. I believe that's what we're observing.

[nicolasbock]

@cwharland Could you create a separate issue? It sounds like it is also caused by #6245 but it's different than the issue with git+ssh URLs. Thanks!

[cjerdonek]

@nicolasbock Thanks again for your quick help with this! 👍

Can you also create a new issue for the slash issue mentioned above, and do you have time to create a PR for that as well? Feel free to let me know if you need advice on the right approach.

[nicolasbock]

No problem @cjerdonek ! I created #6446 .

[K377U]

Thank you for really quick fix!

[cedk]

Do you think there will be soon a bug-fix release with this fix?

[cjerdonek]

Yes, I think so.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.