Implement PEP 685 extra normalization in resolver

[uranusjr]

All extras from user input or dependant package metadata are properly normalized for comparison and resolution. This ensures requests for extras from a dependant can always correctly find the normalized extra in the dependency, even if the requested extra name is not normalized.

Note that this still relies on the declaration of extra names in the dependency's package metadata to be properly normalized when the package is built, since correct comparison between an extra name's normalized and non-normalized forms requires change to the metadata parsing logic, which is only available in packaging 22.0 and up, which pip does not use at the moment.

Close #11649 (but not #11445!)

[pradyunsg]

Yea, this fails to handle packages that don't pull in PEP 685 style extra names. :/

[uranusjr]

Note to self it’d be a good idea to also provide a better error message for this.

[uranusjr]

Alright, addressed the test failure.

In fact I re-implemented this almost entirely, so now the resolver actually keeps track of non-normalized (requested) extras alongside with the normalized form. The normalized form is used in the resolver, except when a candidate inspects distribution metadata to look for dependencies—both the normalized and non-normalized forms are used for look up.

This means that if a package is built without following PEP 685, the user (and the package’s dependants) can still access the extra using the non-normalized form. And once we upgrade to newer packaging with proper PEP 685 support, the non-normalized extras would simply return some duplicated dependencies that are de-duped anyway.

The only failing scenario now would be if a distribution is built without normalizing extras, and is requested with a normalized extra. This would not have worked prior to PEP 685 anyway, and will continue to fail until we upgrade packaging.

[pypa-bot]

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will eligible for code review and hopefully merging!

[uranusjr]

Gosh this setuptools thing is so annoying. I think I’ll try to manually build a package instead 😐

[pradyunsg]

Gosh this setuptools thing is so annoying. I think I’ll try to manually build a package instead 😐

Should be feasible with create_basic_wheel_for_package and foo; extra == 'bar' style dependency declarations.

[RonnyPfannschmidt]

i have a side channel question for this one wrt implementing certain helpers on top of this for normalizing extras of specific versions to a canonical extra (to support default extras)

how hard would it be to expand the resolver to turn extras from a intermediate for to something like "canonical" extras when the solver picks a specific version

(in my wip draft for the pep intend to turn extras like mypkg[foo, -bar] into a canonical form like mypkg[-, foo, ])

if its not unreasonably hard to implement/put on top, i'd go forward with making my draft publishable

[uranusjr]

Shouldn’t be too hard. The point of this is actually to delay marker normalisation as much as possible, and not at all once packaging implements that directly into marker evaluation. The real problem would actually be in pkg_resources inherited from setuptools, which eagerly normalises stuff and necessitates most of the workaround in here. Setuptools is where you’ll get the most issues with or without this PR.