use of PEP 508 URLs dependencies

[benoit-pierre]

The next version of pip is going to support PEP 508 URLs (with use of dependency links being deprecated and slated for removal in 19.0: pypa/pip#4187 (comment).

This works fine when installing from source, but not from wheels, while the issue can easily be fixed in the wheel package, it's unclear to me whether Requires-Dist: name @ url is actually valid 2.1 metadata, as specified in PEP 566.

[pradyunsg]

/cc @di @pfmoore @ncoghlan

[pfmoore]

My feeling is that PEP 566 doesn't allow direct references in Requires-Dist, because it talks about "Version Specifiers", and PEP 440 makes a distinction between Version Specifiers and Direct References (they are described in different sections).

Having said that, I think that the intention is probably that direct references should be allowed in Requires-Dist, as that's the intended replacement for dependency links (as I understand it).

To be honest, though, I don't really understand the workflows that rely on dependency links, so I've no real idea if we're actually solving the underlying issue that they are used for. The whole thing feels a bit like an XY Problem to me.

[di]

The only thing PEP 566 changes that relates to the Requires-Dist field is that specifiers no longer are required to be surrounded in parens as specified in PEP 345:

Requires-Dist: pip (>=19.0)

can now become:

Requires-Dist: pip >= 19.0

Otherwise it's unchanged, so everything else previously specified about the field, including PEP 440-style direct references, should still be considered valid.

[pradyunsg]

@di I don't think 345 allowed direct references - https://www.python.org/dev/peps/pep-0345/#requires-dist-multiple-use

[pfmoore]

Well, given that what PEP 345 says about Requires-Dist points to its own description of a version specifier, that's similar to, but not the same as what PEP 440 says (e.g., ~= isn't supported), and there's nothing about direct references in PEP 345 at all, I think we have to assume it's all a bit of a muddle :-( And as you say, PEP 566 carefully avoided getting sucked into extensive rewrites of that area :-)

But PEP 345 predated PEP 440 by 8 years, so I'm going to stick with my view that somewhere or other, the intention is probably that we should be able to put PEP 440 specifiers (including direct references) into Requires-Dist. But it may well need someone to take on the job of doing a (yet another) revised version of the PEP to make the wording actually say that.

[pradyunsg]

the intention is probably that we should be able to put PEP 440 specifiers (including direct references) into Requires-Dist.

I think so too -- https://packaging.python.org/specifications/core-metadata/#requires-dist-multiple-use references PEP 508 for "full details of the allowed format", which in turn references PEP 440. That'd mean the description given in the guide is merely a subset due to how PEP 345 has defined version specifiers.

But it may well need someone to take on the job of doing a (yet another) revised version of the PEP to make the wording actually say that.

IIUC, we'd want a Metadata v2.2 to extend v2.1 to extend Requires-Dist (and Provides-Dist, Obsoletes-Dist as well) to be defined in terms of PEP 440?

If this is indeed the case, I'd like try my hand at writing the update PEP/RFC/however we're doing this.

[di]

I think the assumption is that PEP 440 is supposed to replace the version specification in PEP 345. From PEP 440:

This document addresses several limitations of the previous attempt at a standardized approach to versioning, as described in PEP 345 and PEP 386.

From PEP 566:

Version numbering requirements and the semantics for specifying comparisons between versions are defined in PEP 440.

Since both are accepted, I take that to mean that PEP 440 is the specification for version specifiers and everything that it defines (including direct references) can be used in any of the fields that can contain version specifiers (including Requires-Dist).

I think a new metadata version and a new PEP are probably unnecessary here, if we really need to we can update the core metadata specs and PEP 566 as needed, since that PEP was supposed to bring in all the changes from PEP 508 and PEP 440 into a new version.

[pradyunsg]

I think a new metadata version and a new PEP are probably unnecessary here, if we really need to we can update the core metadata specs and PEP 566 as needed, since that PEP was supposed to bring in all the changes from PEP 508 and PEP 440 into a new version.

That sounds good and it's a lot less work than a new PEP, so I'm happy to do the writing for this too. :)

[benoit-pierre]

OK, no PEP update necessary sounds good to me too, thanks!

[di]

To be clear, I said that a new PEP (and metadata version) was not necessary. An update to PEP 566 may still be necessary to clear things up, which @pradyunsg volunteered to do.

[benoit-pierre]

Right, do you want me to keep this open to track it?

[ncoghlan]

The intended clause from the withdrawn PEP 426 that should make its way into PEP 566 in some form is this one:

Public index servers MAY prohibit strict version matching clauses or direct references in this field.

That's from https://www.python.org/dev/peps/pep-0426/#dependencies, so it would map to Requires-Dist et al in PEP 566.

The perspective driving the addition of that clause is that public index servers need explicit permission to reject direct references in uploaded metadata because tools are otherwise expected to support them as described in PEP 508.

[di]

@benoit-pierre Yes, let's reopen until PEP 566 is revised.