pip install should install packages in order to avoid ABI incompatibilities in compiled packages

[anntzer]

• Pip version: 9.0.1
• Python version: 3.5
• Operating System: Arch Linux

Description:

pip install foo bar does not list packages in the listed order.

What I've run:

pip install numpy==1.8.0 matplotlib first installs matplotlib, which pulls the most recent numpy as a dependency, builds matplotlib against it, then downgrades to numpy 1.8.0 which has an incompatible ABI, and the matplotlib build is unusable.

Simimar to #3966 but I think the use case above (ABI compatibility) is yet another reason why the listed order of requirements should be respected: earlier requirements may pin the version of dependencies of later packages.

[dstufft]

I don't think that installing packages in order is the right way to solve this, because at some point we're going to get parallel building and then we're still stuck here. I did however, want to link this to #988 and #4144, because I think it points out that we may not be able to completely have build dependencies be isolated like we originally planned.

[piotr-dobrogost]

(...) yet another reason why the listed order of requirements should be respected.

Just a small remark from UX perspective. The common case is that the order should not matter thus by default pip should not obey the order. Respecting the order should be possible to switch on with a flag.

[anntzer]

pip install --respect-order (up to option name bikeshedding) would be fine with me too.

[njsmith]

This is an argument for isolated build dependencies IMO – matplotlib should be listing its build dependency as numpy == <whatever the oldest supported version is>, so if you do pip install matplotlib it pulls matplotlib + an old numpy, builds matplotlib against that, and then installs matplotlib + new numpy.

[dholth]

We have no way to express that every unique build of numpy can have an ABI that affects things built against it. I assume that you could have compiled numpy with options, and it wouldn't matter if you had compiled matplotlib against whatever the oldest supported version of numpy is, it would still be broken.

[njsmith]

We don't have any standardized metadata for this, true. There are two things to improve, I guess: * More expressive ways to describe abi compatibility in install-requires (e.g. ability to say "this wheel depends on numpy-abi-3"). I had a sketch of a design for this at some point. Mostly what you need is a concept of "provides". * Some way for matplotlib's build system to figure out what install-requires metadata to attach to the output wheel. This needs coordination between numpy and matplotlib, but doesn't necessarily need or benefit from standardization. Numpy already has a way to ask where its headers are (np.get_include()); it could have a similar way to ask what its abi requirements are ("np.get_abi_install_requires()").

[dholth]

Third, the traditional option followed by everyone else, conda, dpkg etc., which is to compile A with exactly the same B that will be installed.

[njsmith]

That's not how numpy's ABI compatibility works, though, and for good reason. If it were then we'd have to convince every downstream package to re-release all their wheels every time we made a release.

[dholth]

Good

[njsmith]

Oh, I missed your comment about build options changing the exported ABI. Numpy doesn't have those, because it would make a mess in these situations :-). Don't get me wrong, it would definitely be nice if we had more expressive metadata for expressing these kinds of install requirements (the "provides" idea I gestured vaguely at). But I don't think we need anything else at the PEP level here, at least for now.

[pradyunsg]

I think the best way to handle this is what @dholth says everyone else does:

compile A with exactly the same B that will be installed.

Basically, resolve the versions before starting the builds and use those versions specifically.

[westurner]

#988 "Pip needs a dependency resolver"

[dstufft]

I'm going to close this, we're not going to make order something that people can control. If we need a better mechanism for handling ABI than we should figure out something that explicitly handles that (which would need to take place on the distutils-sig mailing list and likely a PEP).