Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for outputting a list of extras and their requirements. #4824

Open
software-opal opened this issue Oct 30, 2017 · 26 comments
Open
Labels
C: extras Handling optional dependencies C: list/show 'pip list' or 'pip show' state: awaiting PR Feature discussed, PR is needed state: needs discussion This needs some more discussion type: enhancement Improvements to functionality

Comments

@software-opal
Copy link

  • Pip version: 9.0.1
  • Python version: 3.6
  • Operating system: Ubuntu 17.10

Description:

I'm trying to write a script to automatically generate constraints files(with hashes) and I'd like a way to allow this script to detect any extras, and generate the constraints file for those too.

I have no preference to the command; but it might be nice to put it in pip show as another section(depending on compatibility)

What I've run:

$ pip show requests
Name: requests
Version: 2.18.4
...
Requires: idna, urllib3, chardet, certifi
ExtraRequires[security]: cryptography, pyOpenSSL
ExtraRequires[socks]: PySocks
@software-opal
Copy link
Author

An alternative would be to create a pip metadata <package> command to output the <...>.dist-info/METADATA or <...>.dist-info/metadata.json files so that they can be parsed by downstream tooling.

@pradyunsg pradyunsg added state: needs discussion This needs some more discussion type: enhancement Improvements to functionality C: list/show 'pip list' or 'pip show' labels Nov 1, 2017
@pradyunsg
Copy link
Member

This seems to be deeply related and a better alternative to #3797.

@jaraco
Copy link
Member

jaraco commented Nov 1, 2018

An even better alternative would be to use importlib_metadata, which has an API.

>>> import importlib_metadata
>>> importlib_metadata.metadata('xonsh').get_all('Provides-Extra')
['linux', 'mac', 'proctitle', 'ptk', 'pygments', 'win']
>>> importlib_metadata.metadata('xonsh').get_all('Requires-Dist')
["distro; extra == 'linux'", "gnureadline; extra == 'mac'", "setproctitle; extra == 'proctitle'", "prompt-toolkit; extra == 'ptk'", "pygments (>=2.2); extra == 'pygments'", "win-unicode-console; extra == 'win'"]

@jaraco
Copy link
Member

jaraco commented Nov 1, 2018

And use packaging to parse them:

>>> req = next(map(packaging.requirements.Requirement, importlib_metadata('xonsh').get_all('Requires-Dist')))
>>> req.name
'distro'
>>> req.specifier
<SpecifierSet('')>
>>> req.extras
set()
>>> req.marker
<Marker('extra == "linux"')>

@glyph
Copy link

glyph commented Apr 23, 2019

The lack of this feature is an increasingly troublesome annoyance for Twisted. Since python packaging is allegedly Good Now, Actually™ (thanks for that, by the way!), we're starting to rely more on asking our users to install e.g. twisted[tls] or twisted[all_non_platform] in order to get specific optional features, rather than testing huge and ancient dependency matrices so everything will work correctly with default packages on whatever unfortunate end-of-lifed Linux distribution our users happen to be installing on.

However, how is the user supposed to know if it's twisted[tls] or twisted[ssl]? Where do they find out about the string all_non_platform? It would be really great if we could not only get a list of extras, but also like, include some documentation with each one so that as someone is pip install'ing, it's clear to them what extras are available and what each one means.

Thanks for reading!

@pfmoore
Copy link
Member

pfmoore commented Apr 23, 2019

Given that, as @jaraco mentioned, the data is available via importlib_metadata and packaging, maybe someone could write and publish a standalone script to extract this data?

I'm sympathetic to the idea that making the information available via pip aids discoverability, but given that it's possible to get the information the OP needed already, it seems less than ideal to block any progress on this until it can be added to pip. (If nothing else, a standalone script could be the basis of a future PR for pip).

@exarkun
Copy link

exarkun commented Apr 25, 2019

I'm sympathetic to the idea that making the information available via pip aids discoverability, but given that it's possible to get the information the OP needed already, it seems less than ideal to block any progress on this until it can be added to pip.

The OP posted the issue a year and a half ago. I hope they already found a way to not be blocked in their endeavours. :) This is the pip issue tracker though. It seems appropriate to focus on how to support this use-case in pip here.

@exarkun
Copy link

exarkun commented Apr 25, 2019

The lack of this feature is an increasingly troublesome annoyance for Twisted. Since python packaging is allegedly Good Now, Actually™ (thanks for that, by the way!), we're starting to rely more on asking our users to install e.g. twisted[tls] or twisted[all_non_platform] in order to get specific optional features, rather than testing huge and ancient dependency matrices so everything will work correctly with default packages on whatever unfortunate end-of-lifed Linux distribution our users happen to be installing on.

However, how is the user supposed to know if it's twisted[tls] or twisted[ssl]? Where do they find out about the string all_non_platform? It would be really great if we could not only get a list of extras, but also like, include some documentation with each one so that as someone is pip install'ing, it's clear to them what extras are available and what each one means.

Thanks for reading!

One thing to note is that the OP seems to have been discussing gathering this information for programmatic consumption ("I'm trying to write a script to automatically ...") but this comment from @glyph is firmly focused on user education and so probably manual usage by a human. These are both good and interesting use-cases but they may not have the same solution.

@tsvikas
Copy link

tsvikas commented Dec 17, 2019

while it might be a slight different enhancement than what OP meant, I think we all would be glad to have a pip search-extras <package>, that will output the available extras for an (uninstalled) package.
pip is a tool that allows us to list (search) packages & than install them, it allows us to install extras, but how can we discover them?
I believe it's more similar to what @glyph wanted?

Using the
import importlib_metadata; importlib_metadata.metadata('twisted').get_all('Provides-Extra')
requires a venv with importlib_metadata installed, and running python code - not an easy interaction when you are just trying to remember a name of an extra to install.

@pradyunsg pradyunsg added the state: awaiting PR Feature discussed, PR is needed label Feb 6, 2020
@pradyunsg
Copy link
Member

Okay, this feature makes sense and we'd want to include it in pip. We welcome folks to submit a PR for implementing this functionality. Note that this does not mean that the said PR would be accepted - it would still be subject to our regular code reviews as with every other PR in pip.

The relevant bit of code is likely in src/pip/_internal/commands/show.py. Currently, pip uses pkg_resources and the main task is to figure out how to correctly compute the extras a package supports and getting the dependencies that are specified in that extra.

@uranusjr
Copy link
Member

uranusjr commented Feb 6, 2020

There are actually two features this thread talks about.

The top post describes showing showing extras in pip show. The functionality is simple to implement (as demostrated multiple times here), but the interface would require some thought. Personally I would prefer pip to ditch the current pip show formatand introduce something that shows the actual metadata (instead of a transformed version it currently outputs), which would greatly simplies things. Maybe a process like the previously happened pip list migration would be applicable here. If pip decides to stick with the current show format, it would be another problem to come up with a good format to display extras.

But that does not really help the discovery problem mentioned by @glyph, since show only works for installed packages. Currently there are essentially no ways packages could advertise their extras (except document them manually). The first solution that comes to mind is to show that on PyPI like Crates.io (at the bottom in the right-hand column). The problem though is that Python package requirements are declared per artifact, not per version, so thoeratically a package can declare different extras for its Windows and Linux wheels of the same version. This is a very weird things to do in almost all cases, but still it is a thing, and PyPI needs to accomodate that, making the feature difficult to design.

A more workable solution would be to let pip show work with non-installed wheels. The simplese addition would be to allow pip show [path-to-wheel] so people can combine it with pip download to achieve what they want. There’s another possible step pip can take to implement something like pip show --download [name], but that’s probably worth its own discussion.

I hope this thought dump would be marginally helpful for anyone interesting in implementing this feature.

@deveshks
Copy link
Contributor

deveshks commented May 5, 2020

Personally I would prefer pip to ditch the current pip show format and introduce something that shows the actual metadata (instead of a transformed version it currently outputs), which would greatly simplies things.

The main suggestions provided in #8008 (review) which is the documentation PR for #7967 (Adding json format display to pip show) is to add a metadata key in the json output which will have a dictionary containing the fields in https://packaging.python.org/specifications/core-metadata/ . That should be able to take care of listing extras and their requirements atleast for installed packages .

@fabiencelier
Copy link

+1 for listing the extras of a package

We want to split our package into smaller packages linked as extras and it would be nice to see what are all the available extras.

Also I have open a stackoverflow question about that before finding this thread

@davidgilbertson
Copy link

For anyone here looking for something to copy/paste:

from importlib.metadata import metadata
from collections import defaultdict
from pprint import pprint


def get_extras(package_name):
    extras = metadata(package_name).get_all("Provides-Extra")
    required_dists = metadata(package_name).get_all("Requires-Dist")

    result = defaultdict(list)
    for extra in extras:
        for required_dist in required_dists:
            suffix = f" ; extra == '{extra}'"
            if required_dist.endswith(suffix):
                result[extra].append(required_dist.replace(suffix, ""))

    return result


pprint(get_extras("package_name_goes_here"))

This will output the extras, and the packages that each extra will install. E.g. for the transformers package:

{
    "accelerate": ["accelerate (>=0.10.0)"],
    "audio": ["librosa", "pyctcdecode (>=0.3.0)", "phonemizer"],
    "codecarbon": ["codecarbon (==1.2.0)"],
    "deepspeed": ["deepspeed (>=0.6.5)", "accelerate (>=0.10.0)"],
	...
}

I have no idea how robust this is vis-a-vis the layout of the METADATA file, but it Works on My Machine™

@q0w
Copy link
Contributor

q0w commented Nov 1, 2022

@davidgilbertson You can use pip inspect (#11245) with jq

@glyph
Copy link

glyph commented Nov 1, 2022

Something like… pip inspect | jq '.installed[]|select(.metadata.name=="Twisted").metadata.provides_extra' ?

@stonebig
Copy link
Contributor

stonebig commented May 21, 2023

'pip inspect' is ok, and gives pypa devs full liberty to change/improve anything.
An option 'from pip import inspect', to get the same full json string, would be even better, if it could remove the underlying command line legacy

@meejah
Copy link

meejah commented Oct 4, 2023

It would be ideal if an implementation for #7122 had an error-message similar to: There is no extra "foo". This package has extras: "bar", "quux" and "zinga".

@kratsg
Copy link

kratsg commented Oct 20, 2023

Here's a one-liner based on @stonebig's answer that uses jq as well:

$ python -m pip inspect | jq '.installed[] | select(.metadata.name == "scipy").metadata.provides_extra'
WARNING: pip inspect is currently an experimental command. The output format may change in a future release without prior warning.
[
  "test",
  "doc",
  "dev"
]

@Nachtalb
Copy link

Nachtalb commented Nov 3, 2023

Something like… pip inspect | jq '.installed[]|select(.metadata.name=="Twisted").metadata.provides_extra' ?

Here's a one-liner based on @stonebig's answer that uses jq as well:

$ python -m pip inspect | jq '.installed[] | select(.metadata.name == "scipy").metadata.provides_extra'

Built on top of @glyph's and @kratsg's answers I came up with a jq query that filters out all the deps from required_dist and provides them as well:

pip inspect | jq ".installed[] | select(.metadata.name == \"aiogram\").metadata.requires_dist
  | map(select(contains(\"; extra ==\")) | capture(\"(?<dep>.+); extra == (\\\"|')(?<extra>[^']+)(\\\"|')\" ) )
  | group_by(.extra)
  | map( { (.[0].extra) : map(.dep) } )
  | add"
{
  "dev": [
    "mypy",
    "typing_extensions",
    "types-psutil",
    "pycodestyle",
    "ruff",
    "cython-lint>=0.12.2",
    "rich-click",
    "click",
    "doit>=0.36.0",
    "pydevtool"
  ],
  "doc": [
    "sphinx!=4.1.0",
    "pydata-sphinx-theme==0.9.0",
    "sphinx-design>=0.2.0",
    "matplotlib>2",
    "numpydoc",
    "jupytext",
    "myst-nb",
    "pooch"
  ],
  "test": [
    "pytest",
    "pytest-cov",
    "pytest-timeout",
    "pytest-xdist",
    "asv",
    "mpmath",
    "gmpy2",
    "threadpoolctl",
    "scikit-umfpack",
    "pooch"
  ]
}

And if you want to put it in a script and are too lazy to add the necessary bits XD here you go:

#!/usr/bin/env sh
python -m pip inspect | jq ".installed[] | select(.metadata.name == \"$1\").metadata.requires_dist
  | map(select(contains(\"; extra ==\")) | capture(\"(?<dep>.+); extra == (\\\"|')(?<extra>[^']+)(\\\"|')\" ) )
  | group_by(.extra)
  | map( { (.[0].extra) : map(.dep) } )
  | add"

charliermarsh pushed a commit to astral-sh/uv that referenced this issue Mar 12, 2024
## Summary
Follow-up for
395be44

adds `Requires` field to pip show output.

I've aimed to make it behave exactly the same as `pip` does for now, but
there seem to be subtle issues that may require some discussion going
forward:
- Should `uv pip show` support extras? `pip` has an open issue for it,
but currently does not support pypa/pip#4824.
- Relatedly, `Requred-by` field (not implemented in this PR) in `pip
show` currently doesn't take the extras into account transparently, i.e.
when `PySocks` has been installed as an extra for `requests[socks]`,
`pip show PySocks` doesn't have `requests` or `requests[socks]` under
`Requred-by` field. Should `uv pip show` for now just replicate `pip`'s
behavior for now for simplicity and parity or try to cover the extras
for completeness?

## Test Plan
Added a couple of tests:
1. `requests==2.31.0` has four dependencies that would be ordered
differently unless sorted. Additionally, it has two dependencies that
are optionally included for extras.
2. `pandas==2.1.3` depends on different versions of `numpy` depending on
the python version used.
@mierzejk
Copy link

mierzejk commented Apr 5, 2024

I have create a tiny Python gist to print out package extras' requirements: get_package_extras.py.

@uranusjr
Copy link
Member

uranusjr commented Apr 9, 2024

Do we still want a dedicated feature for this, now that pip inspect sort of covers this (and a lot more)?

@mierzejk
Copy link

mierzejk commented Apr 9, 2024

@uranusjr Indeed, all information is available with pip inspect. But the problem is pip inspect returns a huge JSON structure, so if you want to have just a bit of information, like package extras and their requirements, you have to parse and filter what pip inspect returns. If there is a sufficiently great number of people who would find extracting a subset of pip inspect output and filtering it with some parameters useful, I believe implementing such a functionality will be of value.

@stonebig
Copy link
Contributor

stonebig commented Apr 9, 2024

You may close this old thread, and the idea may come back when a fundamental change will happen, like 'uv' integration

@meejah
Copy link

meejah commented Oct 2, 2024

Do we still want a dedicated feature for this, now that pip inspect sort of covers this (and a lot more)?

I do.

Parsing hundreds of lines of JSON is "real work". The most-promising solution (thanks @mierzejk !) is 25 lines of python, not exactly fodder for a bash alias.

I'm here on this thread again in late 2024 because I've searched for how to do this, and been led here from a (now useless) StackOverflow answer. When it's faster to look in the source, understand setup.py and/or pyproject.toml I think this deserves a feature, personally.

I'd be way more willing to take a stab at a PR if a concrete goal was outlined here.

Like: what does the option or subcommand look like?

(For me personally, I'd really like to have something like what #7122 asks for -- both an error and helpful message when installing and unknown extra -- but that thread also doesn't appear to have consensus. Since this has come up for me several times recently, I'd love to have some "pip developer / ux guidance" on what to do here, what might be most-acceptable: a command asking for valid extras, or an error-message like 7122 wants? Does anyone have a few more hints towards what pip developers or other users would find acceptable here?).

For the purposes of spurring discussion, what about:

  • pip list-extras example-package
  • this outputs any valid extras for example-package one line at a time (or nothing if there are no extras)
  • that is if both pip install example-package[foo] and pip install example-package[bar] are valid, it would output:
$ pip list-extras example-package
bar
foo

@fabiencelier
Copy link

For the purposes of spurring discussion, what about:

pip list-extras example-package
this outputs any valid extras for example-package one line at a time (or nothing if there are no extras)
that is if both pip install example-package[foo] and pip install example-package[bar] are valid, it would output:

$ pip list-extras example-package
bar
foo

This 100% what I'm looking for

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C: extras Handling optional dependencies C: list/show 'pip list' or 'pip show' state: awaiting PR Feature discussed, PR is needed state: needs discussion This needs some more discussion type: enhancement Improvements to functionality
Projects
None yet
Development

No branches or pull requests