pip list is unable to mention architecture

[ssbarnea]

Description

If someone happens to upgrade from intel to aarch64, pip will not be able to help him identify any packages that are using wrong architecture. They would have to either uninstall or use install --force in order to address the issue.

Expected behavior

At least the JSON output of pip list should mention architecture when that is avaiable but now it does not do this:

python -m pip list --format=json

Ideally pip should all be smart enough to detect a mismatch in architecture and ignore or warn about mismatched packages. For example pip check would be a good candidate for that too.

pip version

22.0.4

Python version

3.11.3

OS

MacOS

How to Reproduce

It might be harder to reproduce the way I got into it, as I used Apple Migration Assistant to migrate from old computer to the new one but I guess you could just force pip to install a binary wheel using a different architecture.

Once this happens, you are kinda clueless if the wheel is usable or not, at least until you try to do an import from the binary parts.

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

Unfortunately I don’t think there’s metadata recording the wheel architecture available at all, after a wheel is installed. It’s not even theoratically possible, since packages not installed from wheel (but e.g. from setup.py install) doesn’t have that metadata to begin with. So this would need

1. A way to record wheel metadata on installation. Simpliest way is to copy WHEEL-METADATA into the dist-info directory). This may need a discussion in packaging.python.org and a standard (PEP).
2. Remove setup.py install so we always install wheels and get that metadata. Deprecate call to setup.py install when building a wheel failed for source distributions without pyproject.toml #8368 would start the process.

[domdfcoding]

Doesn't the .dist-info/WHEEL file include the wheel tags, from which the architecture can be parsed? I can't see an explicit requirement from PEP 427 that the file be copied over from the wheel, but I think all tools do in practice.

[uranusjr]

Oh I wasn’t aware of that 😆 Someone must have used a time machine. So the question (aside from eliminating setup.py install) would be how to present that message appropriately. Feel free to try it out then.

[ssbarnea]

As I successfully dropped setup.py from all projects I am active with, I am not so concerned about not having support for it. Still, I am quite interested into getting arch info when listing, it could prove very useful. For start maybe we can write an utility script that does this or just add it as a feature to pip check?

[pradyunsg]

I think starting by writing a utility script for this is a good idea -- that makes it possible to demonstrate that this can be implemented and can be immediately usable for the situation you're dealing with.

That can also be adapted into pip's code, if the need to do so still exists after that. :)

[q0w]

packages = get_default_environment().iter_installed_distributions()
for p in packages:
    with open(f"{p.info_location}/WHEEL", 'r') as f:
        tags = list(map(lambda s: parse_tag(s), Parser().parse(f).get_all('Tag')))
        print(tags)

[ssbarnea]

@q0w While trying to test that snipped I was not able to find any class Parser in pip source code.

[q0w]

@ssbarnea its from stdlib email.parser

[ssbarnea]

New version of the script:

#!/usr/bin/env python
from os.path import isfile
from email.parser import Parser
from pip._internal.metadata import get_default_environment
from pip._internal.utils.compatibility_tags import get_supported
from pip._vendor.packaging.tags import parse_tag

supported = get_supported()
packages = get_default_environment().iter_installed_distributions()
for p in packages:
    wheel_file = f"{p.info_location}/WHEEL"
    if isfile(wheel_file):
        with open(wheel_file, 'r', encoding="utf-8") as f:
            # aaa = [parse_tag(x) for x in Parser().parse(f).get_all('Tag')]
            # print(aaa)
            for entry in list(map(parse_tag, Parser().parse(f).get_all('Tag'))):
                for tag in entry:
                    # pylint: disable=protected-access
                    platform = tag._platform
                    if tag in supported or platform == 'any':
                        continue
                    print(f"{p} is using unsupported platform {platform}")

Now, the question is if it would be seen acceptable to include this inside pip check. @pradyunsg WDYT?

Some example from a local run:

$ pip-doctor        
yarl 1.7.2 is using unsupported platform macosx_10_9_x86_64
wrapt 1.14.0 is using unsupported platform macosx_10_9_x86_64
PyYAML 6.0 is using unsupported platform macosx_10_9_x86_64
psutil 5.9.0 is using unsupported platform macosx_10_9_x86_64
multidict 6.0.2 is using unsupported platform macosx_10_9_x86_64
MarkupSafe 2.1.1 is using unsupported platform macosx_10_9_x86_64
lazy-object-proxy 1.7.1 is using unsupported platform macosx_10_9_x86_64
frozenlist 1.3.0 is using unsupported platform macosx_10_9_x86_64
cryptography 36.0.2 is using unsupported platform macosx_10_10_x86_64
coverage 6.2 is using unsupported platform macosx_10_9_x86_64
black 22.3.0 is using unsupported platform macosx_10_9_x86_64
aiohttp 3.8.1 is using unsupported platform macosx_10_9_x86_64

Apparently doing pip uninstall pip install can sort the issue.

[domdfcoding]

For completeness here is a standalone version I made before going on holiday (and getting waylaid with other things on my return).

[uranusjr]

Personally I am +1 to adding this to pip check.

[q0w]

Check only platform tags or also interpreter, abi compatibility?

[uranusjr]

I think we should check all of them, but we don’t necessarily have to implement them all at once.

[sbidoul]

Cross linking #11097 as this makes me reflect on the json output format.

[filbranden]

UPDATE: Reported as #12884

We ran into issues with this new pip check feature in some packages we push.

First three are on Python 3.10 on Linux

• catboost-1.1.1.dist-info/WHEEL lists no Tag's:

Wheel-Version: 1.0
Root-Is-Purelib: false

• xgboost-1.6.1.dist-info/WHEEL has Tag's for cp39-cp39-manylinux_2_17_x86_64 and cp39-cp39-manylinux2014_x86_64, but looking at the list in get_supported() Python 3.10 only lists cp39 with abi3 as the second component, it has cp39-abi3-manylinux_2_17_x86_64 and cp39-abi3-manylinux2014_x86_64 which do not match exactly. Contents of xgboost-1.6.1.dist-info/WHEEL below:

Wheel-Version: 1.0
Generator: bdist_wheel (0.37.1)
Root-Is-Purelib: false
Tag: cp39-cp39-manylinux_2_17_x86_64
Tag: cp39-cp39-manylinux2014_x86_64

• ninja-1.11.1.1.dist-info/WHEEL has a newline above the Tag's, which makes email.parser used in pip return no Tag's since it's expecting no blank lines between header lines:

Wheel-Version: 1.0
Generator: skbuild 0.17.6
Root-Is-Purelib: false

Tag: py2-none-manylinux1_x86_64
Tag: py2-none-manylinux_2_5_x86_64
Tag: py3-none-manylinux1_x86_64
Tag: py3-none-manylinux_2_5_x86_64

• We also encountered this issue with the extra blank line on frozendict-2.3.8.dist-info/WHEEL on a Python 3.11 setup:

Wheel-Version: 1.0
Generator: bdist_wheel (0.40.0)
Root-Is-Purelib: true

Tag: py311-none-any

I understand some of these could be blamed on the packages and how they were built, but it's still unfortunate that we'll start getting pip check warnings for these, so I thought I would report my findings here. (Also, it was not very easy to troubleshoot the issue, essentially I had to reproduce the commands in this PR to understand what was really going on, since there was no useful output or debug logging to help understand the breakage.)

Thank you!