pip fails to build project from backend-path when MetaPathFinder is present and installing from VCS url

[abravalheri]

Description

pip will fail to build a project in the following circumstances:

• A build-backend is installed using a MetaPathFinder (e.g. as if it was installed in editable mode) that is inserted before importlib.machinery.PathFinder.
• The same virtual environment is used to perform an installation from a VCS URL of a project which matches the name of the build-backend previously installed (to a target directory).

Real-world scenario/motivation:

I run into this problem when trying to address pypa/setuptools#3806 and pypa/setuptools#3828. In those issues, users describe problems with the precedence of editable installs with MetaPathFinder. My initial approach was to "bump" the precedence by changing the insert position in sys.meta_path and sys.path_hook.

However, when running the tests with pytest-perf, I received an error message from pip:

pip._vendor.pyproject_hooks._impl.BackendInvalid: Backend was not loaded from backend-path

Note that pytest-perf will install the project from the tracking VCS URL to use as a baseline for the performance comparison.

References

Implementation: pypa/setuptools#3829
Failed tests: https:/pypa/setuptools/actions/runs/4187531113/jobs/7257489290#step:7:646

Expected behavior

pip would build the project in an isolated environment that is not contaminated by the virtual environment where pip is installed. This should be inline with PEP 517 description:

A build frontend SHOULD, by default, create an isolated environment for each build, containing only the standard library and any explicitly requested build-dependencies.

(or at least use importlib.util.spec_from_file_location to ensure the build-backend is loaded from the place specified in pyproject.toml > [build-system] > backend-path)

pip version

23.0.1

Python version

Tested in 3.10.6 and 3.10.9

OS

Tested in Ubuntu 20.04.5 LTS and also python:3.10 container (Debian GNU/Linux 11 - bullseye)

How to Reproduce

1. First we create a dummy backend that uses backend-path

   > docker run --rm -it python:3.10 /bin/bash
   
   rm -rf /tmp/my_backend
   mkdir -p /tmp/my_backend
   cd /tmp/my_backend
   cat <<EOF > pyproject.toml
   [build-system]
   requires = ["flit_core >=3.8.0,<4"]
   backend-path = ["."]
   build-backend = "my_backend"
   
   [project]
   name = "my_backend"
   version = "0.0.0"
   description = "foobar"
   EOF
   
   # Let's simulate a custom build backend ...
   # Assume it has a `build_editable` that uses a MetaPathFinder
   # inserted at position 0 (omitted for the sake of brevity).
   echo 'from flit_core.buildapi import *' > my_backend.py

2. Then we simulate the backend installation in "editable mode" using a MetaPathFinder:

   rm -rf /tmp/.venv
   python3.10 -m venv /tmp/.venv
   /tmp/.venv/bin/python -m pip install -U 'pip==23.0.1'
   /tmp/.venv/bin/python -m pip install 'flit_core >=3.8.0,<4'
   
   # Let's simulate the custom build backend was previously installed in editable mode
   cat <<EOF > /tmp/.venv/lib/python3.10/site-packages/_editable_impl_my_backend.py
   import sys
   from importlib.util import spec_from_file_location
   
   MAPPING = {"my_backend": "/tmp/my_backend/my_backend.py"}
   
   class _EditableFinder:  # MetaPathFinder
       @classmethod
       def find_spec(cls, fullname, path=None, target=None):
           if fullname in MAPPING:
               return spec_from_file_location(fullname, MAPPING[fullname])
   
   def install():
       if not any(finder == _EditableFinder for finder in sys.meta_path):
           sys.meta_path.insert(0, _EditableFinder)
   EOF
   echo 'import _editable_impl_my_backend; _editable_impl_my_backend.install()' > /tmp/.venv/lib/python3.10/site-packages/my_backend.pth
   
   mkdir /tmp/other
   cd /tmp/other
   /tmp/.venv/bin/python -c 'import my_backend; print(my_backend)'
   # => <module 'my_backend' from '/tmp/my_backend/my_backend.py'>

3. Finally, we try to install a different version of the same backend from a VCS URL - which will result in an error:

    rm -rf /tmp/install_target
    /tmp/.venv/bin/python -m pip -vvv install -t /tmp/install_target "my_backend @ git+https://gist.github.com/4f1f86d879179e2d185b6e82e4f50d17.git"
    # ...
    # pip._vendor.pyproject_hooks._impl.BackendInvalid: Backend was not loaded from backend-path

Please note that the backend is not really invalid in this case... What happens is that the build frontend is loading it from a different location instead of loading it from backend-path.

Output

# docker run --rm -it python:3.10 /bin/bash
$ rm -rf /tmp/my_backend
$ mkdir -p /tmp/my_backend
$ cd /tmp/my_backend
$ cat <<EOF > pyproject.toml
[build-system]
requires = ["flit_core >=3.8.0,<4"]
backend-path = ["."]
build-backend = "my_backend"

[project]
name = "my_backend"
version = "0.0.0"
description = "foobar"
EOF
$ echo 'from flit_core.buildapi import *' > my_backend.py
$ rm -rf /tmp/.venv
$ python3.10 -m venv /tmp/.venv
$ /tmp/.venv/bin/python -m pip install -U 'pip==23.0.1'
Collecting pip==23.0.1
  Downloading pip-23.0.1-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 1.2 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.2.1
    Uninstalling pip-22.2.1:
      Successfully uninstalled pip-22.2.1
Successfully installed pip-23.0.1
$ /tmp/.venv/bin/python -m pip install 'flit_core >=3.8.0,<4'
Collecting flit_core<4,>=3.8.0
  Downloading flit_core-3.8.0-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 1.1 MB/s eta 0:00:00
Installing collected packages: flit_core
Successfully installed flit_core-3.8.0
$ cat <<EOF > /tmp/.venv/lib/python3.10/site-packages/_editable_impl_my_backend.py
import sys
from importlib.util import spec_from_file_location

MAPPING = {"my_backend": "/tmp/my_backend/my_backend.py"}

class _EditableFinder:  # MetaPathFinder
    @classmethod
    def find_spec(cls, fullname, path=None, target=None):
        if fullname in MAPPING:
            return spec_from_file_location(fullname, MAPPING[fullname])

def install():
    if not any(finder == _EditableFinder for finder in sys.meta_path):
        sys.meta_path.insert(0, _EditableFinder)
EOF
$ echo 'import _editable_impl_my_backend; _editable_impl_my_backend.install()' > /tmp/.venv/lib/python3.10/site-packages/my_backend.pth
$ mkdir /tmp/other
$ cd /tmp/other
$ /tmp/.venv/bin/python -c 'import my_backend; print(my_backend)'
<module 'my_backend' from '/tmp/my_backend/my_backend.py'>
$ rm -rf /tmp/install_target
$ /tmp/.venv/bin/python -m pip -vvv install -t /tmp/install_target "my_backend @ git+https://gist.github.com/4f1f86d879179e2d185b6e82e4f50d17.git"
Using pip 23.0.1 from /tmp/.venv/lib/python3.10/site-packages/pip (python 3.10)
Non-user install due to --prefix or --target option
Created temporary directory: /tmp/pip-target-rb_hedkt
Created temporary directory: /tmp/pip-build-tracker-vpe84uob
Initialized build tracking at /tmp/pip-build-tracker-vpe84uob
Created build tracker: /tmp/pip-build-tracker-vpe84uob
Entered build tracker: /tmp/pip-build-tracker-vpe84uob
Created temporary directory: /tmp/pip-install-rjiqcf3e
Created temporary directory: /tmp/pip-ephem-wheel-cache-thvjcdal
Collecting my_backend@ git+https://gist.github.com/4f1f86d879179e2d185b6e82e4f50d17.git
  Cloning https://gist.github.com/4f1f86d879179e2d185b6e82e4f50d17.git to /tmp/pip-install-rjiqcf3e/my-backend_90e570b192394d7287e6c771158ffeb3
  Running command git version
  git version 2.30.2
  Running command git clone --filter=blob:none --verbose --progress https://gist.github.com/4f1f86d879179e2d185b6e82e4f50d17.git /tmp/pip-install-rjiqcf3e/my-backend_90e570b192394d7287e6c771158ffeb3
  Cloning into '/tmp/pip-install-rjiqcf3e/my-backend_90e570b192394d7287e6c771158ffeb3'...
  POST git-upload-pack (164 bytes)
  POST git-upload-pack (272 bytes)
  remote: Enumerating objects: 2, done.
  remote: Counting objects:  50% (1/2)
  remote: Counting objects: 100% (2/2)
  remote: Counting objects: 100% (2/2), done.
  remote: Compressing objects:  50% (1/2)
  remote: Compressing objects: 100% (2/2)
  remote: Compressing objects: 100% (2/2), done.
  remote: Total 2 (delta 0), reused 2 (delta 0), pack-reused 0
  Receiving objects:  50% (1/2)
  Receiving objects: 100% (2/2)
  Receiving objects: 100% (2/2), done.
  Running command git rev-parse HEAD
  1577d5b524e74766a2b9b8cf6a7cc73e630794e7
  Resolved https://gist.github.com/4f1f86d879179e2d185b6e82e4f50d17.git to commit 1577d5b524e74766a2b9b8cf6a7cc73e630794e7
  Running command git rev-parse HEAD
  1577d5b524e74766a2b9b8cf6a7cc73e630794e7
  Added my_backend@ git+https://gist.github.com/4f1f86d879179e2d185b6e82e4f50d17.git from git+https://gist.github.com/4f1f86d879179e2d185b6e82e4f50d17.git to build tracker '/tmp/pip-build-tracker-vpe84uob'
  Created temporary directory: /tmp/pip-build-env-wbowle8_
  Running command pip subprocess to install build dependencies
  Using pip 23.0.1 from /tmp/.venv/lib/python3.10/site-packages/pip (python 3.10)
  Collecting flit_core<4,>=3.8.0
    Using cached flit_core-3.8.0-py3-none-any.whl (62 kB)
  Installing collected packages: flit_core
  Successfully installed flit_core-3.8.0
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Getting requirements to build wheel ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 160, in exc_logging_wrapper
    status = run_func(*args)
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 247, in wrapper
    return func(self, options, args)
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/commands/install.py", line 419, in run
    requirement_set = resolver.resolve(
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 491, in collect_root_requirements
    req = self._make_requirement_from_install_req(
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 453, in _make_requirement_from_install_req
    cand = self._make_candidate_from_link(
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 297, in __init__
    super().__init__(
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 162, in __init__
    self.dist = self._prepare()
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 231, in _prepare
    dist = self._prepare_distribution()
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 308, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 491, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 577, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/operations/prepare.py", line 69, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/distributions/sdist.py", line 48, in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/distributions/sdist.py", line 118, in _install_build_reqs
    build_reqs = self._get_build_requires_wheel()
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/distributions/sdist.py", line 95, in _get_build_requires_wheel
    return backend.get_requires_for_build_wheel()
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_internal/utils/misc.py", line 701, in get_requires_for_build_wheel
    return super().get_requires_for_build_wheel(config_settings=cs)
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
  File "/tmp/.venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 323, in _call_hook
    raise BackendInvalid(
pip._vendor.pyproject_hooks._impl.BackendInvalid: Backend was not loaded from backend-path
Remote version of pip: 23.0.1
Local version of pip:  23.0.1
Was pip installed by pip? True
Removed my_backend@ git+https://gist.github.com/4f1f86d879179e2d185b6e82e4f50d17.git from git+https://gist.github.com/4f1f86d879179e2d185b6e82e4f50d17.git from build tracker '/tmp/pip-build-tracker-vpe84uob'
Removed build tracker: '/tmp/pip-build-tracker-vpe84uob'

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[abravalheri]

By adding some debug statements to pip/_vendor/pyproject_hooks/_in_process/_in_process.py (around line 86), I can see the following:

obj.__file__ = '/tmp/my_backend/my_backend.py'
extra_pathitems = ['/tmp/pip-install-akmokyzj/my-backend_6731a91a030b414da7bf6d41b4d958c8']
sys.path_hooks = [
	<class 'zipimport.zipimporter'>,
	<function FileFinder.path_hook.<locals>.path_hook_for_FileFinder at 0x7f31cda1b910>
]
sys.meta_path = [
	<class '_editable_impl_my_backend._EditableFinder'>,
	<_distutils_hack.DistutilsMetaFinder object at 0x7f31cd921f30>,
	<class '_frozen_importlib.BuiltinImporter'>,
	<class '_frozen_importlib.FrozenImporter'>,
	<class '_frozen_importlib_external.PathFinder'>
]
sys.path = [
	'/tmp/pip-install-akmokyzj/my-backend_6731a91a030b414da7bf6d41b4d958c8',
	'/tmp/.venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process',
	'/tmp/pip-build-env-bwdvn95z/site',
	'/usr/local/lib/python310.zip',
	'/usr/local/lib/python3.10',
	'/usr/local/lib/python3.10/lib-dynload',
	'/tmp/pip-build-env-bwdvn95z/overlay/lib/python3.10/site-packages',
	'/tmp/pip-build-env-bwdvn95z/normal/lib/python3.10/site-packages'
]

[pfmoore]

First off, I will say that this seems to me like a very unusual case. It does look like a bug, but I wouldn't consider it particularly high priority. I want to establish this up front because (see below) it seems rather hard to fix properly.

This might be fixable by adding some extra logic to the injected sitecustomize to wipe out sys.meta_path, etc. But frankly, this starts to feel like we're just playing "whack-a-mole" with ways the user can mess with the import system. IMO, if we want to fix this properly, continuing down this route is likely just a drain on our time, and on the maintainability of the isolated build code. We also need to be clear how we identify what's a "legitimate" hook and what should be stripped - there are 2 path hooks and 5 meta_path hooks in the provided example, only one of which was added as part of the demonstration...

And of course, we need to establish precisely what we're fixing here. "Use of import hooks like the way the editables project currently implements mappings" is way too specific, and means that if editables changes, or someone uses a different mechanism for implementing editable installs, we risk being back to square one (see python/cpython#102097 for an example of how that could happen).

At this point, I think that we'd be better just dumping the existing build environment approach and using venv. We can create a completely empty venv (without pip) and inject the currently running pip like we do at the moment. That should be fast (most of the time it takes to create a venv comes from installing pip). We'd need to consider how we want to deal with environments that don't have venv available - the ones I can think of are the "embeddable distribution" of Python (where people probably shouldn't be using pip anyway), and (I believe) the system Python in some Debian-based distros, which ship venv as an "add on" package. I'm inclined to say that it's OK to take the stance that pip expects venv to be present, but it is a potential breakage.

[abravalheri]

Hi @pfmoore, thank you very much for having a look on this.

I completely agree that it is a very specific scenario, and tricky to solve. Probably not high-priority either.
My intention was just to report and discuss if there is anything that can be done... Maybe that is a limitation that we have to live with.

And of course, we need to establish precisely what we're fixing here. "Use of import hooks like the way the editables project currently implements mappings" is way too specific, and means that if editables changes, or someone uses a different mechanism for implementing editable installs.

If there is an interest from pip's side on support this use case, I think the "fixing" or "improvement" that I would be after, is for pip to create an environment that has the same level of isolation as a brand new environment created with python -m venv or virtualenv. This includes not inheriting any sys.path, sys.path_hooks or sys.meta_path manipulations from the parent environment. (This, in my opinion, would be a good interpretation of the PEP 517 recommendation).

I think that your suggestion of dumping the existing build environment approach and using venv would be a good approach.

[abravalheri]

If the final conclusion is that this is a limitation we have to live with, would it be possible to at least drop the backend-path check? This is the check that halts the build.

Another possibility would be for the frontend to actively import the backend from the right path by using importlib.util.spec_from_file_location or importlib.machinery.PathFinder.find_spec, instead of relying that importlib.import_module will always do the right thing...

[pfmoore]

OK, I'll go on record as in favour of using venv. And not trying to fix up the existing approach. But let's see what the other maintainers think.

I do think that if we do that, we may have a fight on our hands around Debian not shipping venv as a default part of the system Python¹. But I think we should make that Debian's problem - if venv isn't present, fail with a big message saying "your Python installation is incomplete. You should report this issue to your distributor (and in the meantime, look for a "python venv" package distributed by your OS to install)".

Or we could use venv by default but keep the existing apprach as a fallback (without trying to fix this bug in it - there are limits!) But I'd be very sad if we did that, as it's to an extent the worst of both worlds.

Footnotes

1. Assuming we haven't already had that fight. I know it's come up before, but I think we've sidestepped it rather than facing it head on. ↩

[dstufft]

I'm +1 on using venv.

I think that Debian recommends the installation of venv with pip anyways, and there's some effort being done to try and make the disparate use cases better between system stuff and developer stuff. I don't recall what that is offhand (python-full package maybe?).

An interesting thing, one of the reasons (and maybe the reason) that debian doesn't ship venv is due to the fact that venv installs pip, which does a whole thing on Debian. If we ever manage to get to a world where we don't install pip into every random virtualenv and we just expect it to be available on the system and you "target' the virtual env, that might fix that divergence too :)

[pradyunsg]

#11619

🙃

[pfmoore]

Apologies, I should have remembered that now @pradyunsg is a core developer, he has access to the keys to Guido's time machine 😉

[FFY00]

FYI, we've merged a build.env refactor in pypa/build, I haven't personally looked very close at it bc anxiety and lots of stuff to do, but it may now be suitable for pip, if you want to take a look.

[pradyunsg]

Oh nice! I'll take a look at it after spending some bandwidth on the resolver though!

[abravalheri]

I opened a PR on pypa/pyproject-hooks#165 to improve the way backend-path is handled and avoid errors caused by "imperfectly" isolated environments.

My understanding is that pyproject-hooks' improvement and the decision of pip adopting venv-based isolation are somewhat orthogonal/independent, since PEP 517 does not strictly mandate venv-level isolation (moreover, pyproject-hooks error is currently pretty hard to reason about; such change would prevent accidental errors and improve debug-ability).