sdist installation fails when pip is run from wheel

[abn]

Description

This might not be a bug per say as pip is not officially zip safe. However, a previously working scenario is no longer working, figured I would raise this here.

In pip versions prior to 21.1.1, we were able to use pip as shown below.

curl -sLO https://files.pythonhosted.org/packages/fe/ef/60d7ba03b5c442309ef42e7d69959f73aacccd0d86008362a681c4698e83/pip-21.0.1-py3-none-any.whl
python pip-21.0.1-py3-none-any.whl/pip install https://files.pythonhosted.org/packages/52/7e/6a41d1504225ef9213731e69ecebfa8fe78c614bcb01e1147f6242073636/black-21.5b0.tar.gz

However, with 21.1.1 this no longer works.

Expected behavior

An sdist package can be installed successfully when installed with python pip-<version>.whl/pip install <sdist>.

pip version

21.1.1

Python version

3.9

OS

Fedora 34

How to Reproduce

The following commands reproduce the issue.

curl -sLO https://files.pythonhosted.org/packages/cd/6f/43037c7bcc8bd8ba7c9074256b1a11596daa15555808ec748048c1507f08/pip-21.1.1-py3-none-any.whl
python pip-21.1.1-py3-none-any.whl/pip --verbose install https://files.pythonhosted.org/packages/52/7e/6a41d1504225ef9213731e69ecebfa8fe78c614bcb01e1147f6242073636/black-21.5b0.tar.gz

Output

Using pip 21.1.1 from //pip-21.1.1-py3-none-any.whl/pip (python 3.9)
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-ephem-wheel-cache-okb9brsr
Created temporary directory: /tmp/pip-req-tracker-ga2kbjwp
Initialized build tracking at /tmp/pip-req-tracker-ga2kbjwp
Created build tracker: /tmp/pip-req-tracker-ga2kbjwp
Entered build tracker: /tmp/pip-req-tracker-ga2kbjwp
Created temporary directory: /tmp/pip-install-tbgi_mxf
Collecting https://files.pythonhosted.org/packages/52/7e/6a41d1504225ef9213731e69ecebfa8fe78c614bcb01e1147f6242073636/black-21.5b0.tar.gz
  Created temporary directory: /tmp/pip-req-build-remwfpjm
  Created temporary directory: /tmp/pip-unpack-92wlal61
  Looking up "https://files.pythonhosted.org/packages/52/7e/6a41d1504225ef9213731e69ecebfa8fe78c614bcb01e1147f6242073636/black-21.5b0.tar.gz" in the cache
  No cache entry available
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/52/7e/6a41d1504225ef9213731e69ecebfa8fe78c614bcb01e1147f6242073636/black-21.5b0.tar.gz HTTP/1.1" 200 1152691
  Downloading black-21.5b0.tar.gz (1.2 MB)
  Ignoring unknown cache-control directive: immutable
  Updating cache with response from "https://files.pythonhosted.org/packages/52/7e/6a41d1504225ef9213731e69ecebfa8fe78c614bcb01e1147f6242073636/black-21.5b0.tar.gz"
  Caching due to etag
  Added https://files.pythonhosted.org/packages/52/7e/6a41d1504225ef9213731e69ecebfa8fe78c614bcb01e1147f6242073636/black-21.5b0.tar.gz to build tracker '/tmp/pip-req-tracker-ga2kbjwp'
  Created temporary directory: /tmp/pip-build-env-4zs5x6tm
  Created temporary directory: /tmp/pip-standalone-pip-kf46cir8
  Installing build dependencies: started
  Running command /usr/local/bin/python /tmp/pip-standalone-pip-kf46cir8/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-4zs5x6tm/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel
  /usr/local/bin/python: can't find '__main__' module in '/tmp/pip-standalone-pip-kf46cir8/__env_pip__.zip/pip'
  Installing build dependencies: finished with status 'error'
WARNING: Discarding https://files.pythonhosted.org/packages/52/7e/6a41d1504225ef9213731e69ecebfa8fe78c614bcb01e1147f6242073636/black-21.5b0.tar.gz. Command errored out with exit status 1: /usr/local/bin/python /tmp/pip-standalone-pip-kf46cir8/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-4zs5x6tm/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.
ERROR: Command errored out with exit status 1: /usr/local/bin/python /tmp/pip-standalone-pip-kf46cir8/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-4zs5x6tm/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/cli/base_command.py", line 180, in _main
    status = self.run(options, args)
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/cli/req_command.py", line 204, in wrapper
    return func(self, options, args)
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/commands/install.py", line 318, in run
    requirement_set = resolver.resolve(
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/resolution/resolvelib/resolver.py", line 103, in resolve
    r = self.factory.make_requirement_from_install_req(
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/resolution/resolvelib/factory.py", line 433, in make_requirement_from_install_req
    raise self._build_failures[ireq.link]
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/resolution/resolvelib/factory.py", line 200, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/resolution/resolvelib/candidates.py", line 306, in __init__
    super().__init__(
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/resolution/resolvelib/candidates.py", line 151, in __init__
    self.dist = self._prepare()
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/resolution/resolvelib/candidates.py", line 234, in _prepare
    dist = self._prepare_distribution()
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/resolution/resolvelib/candidates.py", line 317, in _prepare_distribution
    return self._factory.preparer.prepare_linked_requirement(
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/operations/prepare.py", line 570, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/operations/prepare.py", line 60, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/distributions/sdist.py", line 34, in prepare_distribution_metadata
    self._setup_isolation(finder)
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/distributions/sdist.py", line 62, in _setup_isolation
    self.req.build_env.install_requirements(
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/build_env.py", line 199, in install_requirements
    self._install_requirements(
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/build_env.py", line 247, in _install_requirements
    call_subprocess(args, spinner=spinner, extra_environ=extra_environ)
  File "//pip-21.1.1-py3-none-any.whl/pip/_internal/utils/subprocess.py", line 244, in call_subprocess
    raise InstallationSubprocessError(proc.returncode, command_desc)
pip._internal.exceptions.InstallationSubprocessError: Command errored out with exit status 1: /usr/local/bin/python /tmp/pip-standalone-pip-kf46cir8/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-4zs5x6tm/overlay --no-warn-script-location -v --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=41.0' setuptools-scm wheel Check the logs for full command output.
1 location(s) to search for versions of pip:
* https://pypi.org/simple/pip/
Fetching project page and analyzing links: https://pypi.org/simple/pip/
Getting page https://pypi.org/simple/pip/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/pip/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/pip/ HTTP/1.1" 200 16359
Updating cache with response from "https://pypi.org/simple/pip/"
Caching due to etag
  Found link https://files.pythonhosted.org/packages/3d/9d/1e313763bdfb6a48977b65829c6ce2a43eaae29ea2f907c8bbef024a7219/pip-0.2.tar.gz#sha256=88bb8d029e1bf4acd0e04d300104b7440086f94cc1ce1c5c3c31e3293aee1f81 (from https://pypi.org/simple/pip/), version: 0.2
<snipped>
  Found link https://files.pythonhosted.org/packages/94/b0/e10bdc8809c81796c80aa3644a8e3dc16594fb1bd68f5996929f26cad980/pip-21.1.1.tar.gz#sha256=51ad01ddcd8de923533b01a870e7b987c2eb4d83b50b89e1bf102723ff9fed8b (from https://pypi.org/simple/pip/) (requires-python:>=3.6), version: 21.1.1
Skipping link: not a file: https://pypi.org/simple/pip/
Given no hashes to check 165 links for project 'pip': discarding no candidates
WARNING: You are using pip version 20.2.4; however, version 21.1.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
Removed https://files.pythonhosted.org/packages/52/7e/6a41d1504225ef9213731e69ecebfa8fe78c614bcb01e1147f6242073636/black-21.5b0.tar.gz from build tracker '/tmp/pip-req-tracker-ga2kbjwp'
Removed build tracker: '/tmp/pip-req-tracker-ga2kbjwp'

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

Thanks for the report. This is caused by “PEP 517 inception” (a term I invented for #9878), but slightly more obscure. Starting with 21.1, pip now builds itself as a zip file when being used to populate an isolated environment, but when it can’t do that if the “host” pip is already a zip. This is currently detected by checking if pip.__file__ is a zip file. But pip-21.1.1-py3-none-any.whl/pip is not a zip file! So the check needs some fix to account for this (e.g. go up the path chain and see if any of the parents is a file? I didn’t actually check the current implementation and don’t know what the exact right solution is yet.)

I’ll spend some time looking into this in the coming weekend. Adding this into the 21.1.2 milestone as a regression.

[pfmoore]

+1 to fixing this if it's straightforward to do so, and I think it's fair in that context to view this as a regression, but to be clear, as the OP said, this isn't formally supported usage of pip.

Although @uranusjr does the "PEP 517 inception" mechanism (which I haven't looked at too closely) rely on pip-running-from-a-zip being able to install? And if so, does it need to be able to install from general sources (e.g. PyPI)? Because if we're using pip from a zip internally, maybe we need to document that we mustn't break that usage (at a minimum in an "internal use only" section of the docs...)

[abn]

@uranusjr I have attempted a fix (#9960) as it seemed fairly trivial. I have tested it locally for the reported use case and it works. Happy for it to be closed in favour of a more complete one as required.