Add support for command plugins

[msabramo]

This introduces a new concept to pip called "command plugins". This allows a user to implement new features but do them outside pip itself. This means that people can do interesting experimental and/or custom features and implement these features outside of pip so that it doesn't clutter up the core code of pip and it doesn't require the time of pip maintainers to review it.

This might also be an alternative to simply ripping out features of pip that maybe aren't used a ton and are deemed not worth the effort to maintain them (e.g.: pip [un]zip, pip bundle, etc.). Perhaps even pip wheel could be made into a plugin (although I feel that it is probably used much more than the previously mentioned commands, but maybe since pip wheel requires the wheel package anyway, maybe it makes sense to just have the wheel package implement a pip plugin?). Such features could be removed from pip itself, but put into a separate plugin package and maintained by a 3rd party, so that they are not a time burden on PyPA developers.

To create a plugin command, simply:

• Create a third-party Python distribution. It does not have to be on PyPI. It just needs to be installable.
• Create a class that subclasses the pip.basecommand.Command class (example).
• Register your new class with the new pip.command.v1 entry point in the setup.py of your distribution (example).

An example plugin is available at https:/msabramo/pipdeptree/tree/pip_plugin -- in this branch I've forked an already existing pipdeptree project that implements a separate pipdeptree command -- in my fork I've put in the necessary bits so that pipdeptree is a pip plugin.

Showing how it works...

[marca@marca-mac2 ~WORKON_HOME]$ mktmpenv
New python executable in tmp-4adc64ecf8c6ac9e/bin/python
Please make sure you remove any previous custom paths from your /Users/marca/.pydistutils.cfg file.
Installing setuptools, pip...done.
This is a temporary environment. It will be deleted when you run 'deactivate'.

[marca@marca-mac2 ~VIRTUAL_ENV]$ pip install https:/msabramo/pip/archive/command_plugins.zip
...
Successfully installed pip-6.1.0.dev0

# Install the pipdeptree plugin
[marca@marca-mac2 ~VIRTUAL_ENV]$ pip install https:/msabramo/pipdeptree/archive/pip_plugin.zip
Collecting https:/msabramo/pipdeptree/archive/pip_plugin.zip
  Downloading https:/msabramo/pipdeptree/archive/pip_plugin.zip
     - 20kB 85kB/s
Requirement already satisfied (use --upgrade to upgrade): pip>=1.4.1 in ./lib/python2.7/site-packages (from pipdeptree==0.4.2)
Installing collected packages: pipdeptree
  Running setup.py install for pipdeptree
    Installing pipdeptree script to /Users/marca/python/virtualenvs/tmp-4adc64ecf8c6ac9e/bin
Successfully installed pipdeptree-0.4.2

[marca@marca-mac2 ~VIRTUAL_ENV]$ pip

Usage:
  pip <command> [options]

Commands:
  ...
  deptree                     Display a dependency tree of the installed python packages

[marca@marca-mac2 ~VIRTUAL_ENV]$ pip deptree --help

Usage:
  pip deptree [options]

Deptree Options:
  -f, --freeze                Print names so as to write freeze files
  -a, --all                   list all deps at top level
  -l, --local-only            If in a virtualenv that has global access donot show globally installed packages
  -w, --nowarn                Inhibit warnings about possibly confusing packages
...

# Just installing a few packages so that the output of `pip deptree` is more interesting...
[marca@marca-mac2 ~VIRTUAL_ENV]$ pip install flask ipython
...

[marca@marca-mac2 ~VIRTUAL_ENV]$ pip deptree
wsgiref==0.1.2
Flask==0.10.1
  - Werkzeug [required: >=0.7, installed: 0.9.6]
  - Jinja2 [required: >=2.4, installed: 2.7.3]
    - MarkupSafe [installed: 0.23]
  - itsdangerous [required: >=0.21, installed: 0.24]
ipython==2.3.1
  - gnureadline [installed: 6.3.3]
wheel==0.24.0

Cc: @dstufft, @pfmoore, @naiquevin, @therealprologic

Refs: tox-dev/pipdeptree#1

[pfmoore]

Hmm, last time this was discussed, using entry points was an issue because it added a dependency on setuptools (which is problematic because tools like virtualenv use pip without setuptools to install from wheels only). But now we vendor pkg_resources that issue's gone away. The resulting code looks nice and simple.

Some comments:

1. The change could do with docs and tests.
2. 3rd party plugins would depend on pip internals, which could easily break even in a point release (we don't provide a stable API). So there's a risk in using plugins. Is this a problem? (Personally, I'd be tempted to say we just need to document the issue and leave it at that).

[dstufft]

My opinion on allowing to register plugins is basically that unless we provide an API to pip itself the only real benefits is that a user can type something like pip foo instead of pip-foo and perchance it would show up if you did pip -h. This isn't much code, however I question whether the "could easily break even in a point release" part makes it worthwhile at all given that the benefits of using a pip plugin are currently fairly low.

[qwcode]

similar to what I did last year, but we didn't merge: #1409

[msabramo]

I guess I kind of see @dstufft's point that as-is, this doesn't give you a ton of new functionality.

But I do think there's some value in being able to unify stuff. E.g.: to type pip -h and see all of the packaging related commands in one place.

The interesting question here is what API should pip provide. I think pip has seen itself as a tool rather than an API so the API has never been a focus and there are not a ton of tools that I know of that try to use pip -- with pipdeptree being an exception. If it doesn't have a great API then there won't be a lot of tools that use it, regardless of whether there is a plugin mechanism or not. If there is a good API, then folks may be more likely to do interesting things with it and if folks do interesting things with it, then there would be more motivation to make the API stable and then there is not as much risk of breaking things.

It is a different focus for sure. Historically, pip has done everything it wanted to do itself. If time is invested in APIs, then it enables a wider set of people to do features and to move more quickly.

APIs are hard though for sure. And I don't yet know what these APIs would look like.

[qwcode]

beyond the command entry point itself, many plugins would most likely need additional entry points to plug in additional types or steps in various places.

If it were just me, I'd be inclined to just start adding extension points with no promises and see where it went.

[pfmoore]

Having just looked at writing a simple pip-query command, it does seem to me that a plugin system is worth it. It allows you to copy one of pip's built in commands and get argument handling for free, as well as being able to concentrate on your command's logic rather than UI stuff.

We should make it 100% clear that because we allow people to write plugins that use the internal API, does not mean that we guarantee any sort of stability, but that's fine.

It will encourage people to experiment externally with new command suggestions (there are a few such issues open at the moment) and the resulting code will be that much easier to include in core pip should we later decide to do so, just because it will use the standard command conventions.

[pfmoore]

OK, it's definitely worth it IMO. Took me 30 minutes to write a new command by cloning an existing one. I'd previously tried to write a standalone script, and gave up pretty fast having done nothing but try to get basic arg parsing working, working out how to set up a finder with a suitable session. I never even got close to supporting options like --index-url. Being able to write plugins is a huge win, even if only in these sorts of admin costs.

Let's just do it and see how it goes.

@msabramo Could you fix the pep8 failures and add some minimal docs?

[dstufft]

So one other thing that might make us want to hold off is that I'm currently experimenting using click in the virtualenv rewrite. If it works out well there then I think it's reasonable to switch pip itself to use click. If we do that then that's going to be a major break in the API of creating a command right around the time we release the entrypoint which seems less than ideal (even if technically allowed).

[pfmoore]

Hmm, possibly. On the fifteenth hand (!) a 3rd party plugin (and even more so a personal one) can be less "perfect" than a pip builtin command. So, for example, I have knocked up a "pip query" command that I'd really like to try out - the output isn't perfect and there are a lot of rough edges yet, but installing a plugin while I work on that is much easier than using a personal build of pip.

We obviously have different opinions, so it'd be good if others chimed in. @pypa/pip-developers anyone else have a view?

[dstufft]

Well I don't really mean that we shouldn't add it (I don't feel strongly enough to say absolutely not) just that if we're going to add it we might want to hold off until we see if click works out for virtualenv and if we're going to switch pip to it so that we don't go "hey you can make pip plugins!" and then immediately turn around and break anyone who uses that by making a major change to the command infrastructure.

[pfmoore]

Yeah, same for me the other way round, it's not super-urgent as it's no benefit till it's in a released pip anyway. It sounds like the virtualenv rewrite (and hence a decision on click) is going pretty well, so it's probably not as far off as I was assuming. No need to panic about a decision.

[msabramo]

I am using click for some internal tools at work and I love it! So I'd be up for delaying this for that 😄

[msabramo]

Just pushed an update just to fix PEP 8 and rebase on origin/develop.

Has anyone tried playing with click yet? I didn't see a branch for it, but maybe someone is playing with it locally?

[msabramo]

I just played with click a bit in the context of pip.

I must say that although I have loved working with click on a few recent green field projects, I feel like click is probably not a great fit for an existing tool like pip.

I attempted to convert the wheel command over to using click. I had some very limited success in getting the pip optparse code to introspect and call a function defined with a click decorator, but it took quite a while and it's still not working and doesn't feel noticeably better than the current code. Converting a bunch of cmd_opts.add_option calls to @click.option decorators doesn't seem like a big win to me.

[msabramo]

I took a different approach with click that worked a little better than what I was doing before. It seems to work a bit better, but I'm still not convinced that it brings enough value to be worth it.

#2419

[ncoghlan]

I think if there's one lesson to learn from distutils, it's that subclassing based plugin APIs are not a good way to go :)

An approach I've seen (and used) that provides a much cleaner separation of concerns is dependency injection: the containing applications provides the scaffolding to run the plugin, and then passes the plugin an API object that defines the supported interface. This has the combined benefit of making plugins easier to test (since you can pass in a mock API object), as well as making it easier to evolve the API exposed to plugins (since as long as plugins are versioned, you can ensure they receive the correct kind of API object).

And this is where I think switching to click could be valuable in terms of offering a more sustainable plugin API: http://click.pocoo.org/5/complex/#complex-applications

The key there is the support that click has for managing command contexts. While the example in the click docs focuses on how that can be used to pass context to subcommands, and https://pypi.python.org/pypi/click-plugins supports adding new subcommands that parallel existing ones, the context concept can also be reused for passing information and APIs to plugins.

[dstufft]

Accidentally closed this, reopening. Sorry!

[BrownTruck]

Hello!

As part of an effort to ease the contribution process and adopt a more standard workflow pip has switched to doing development on the master branch. However, this Pull Request was made against the develop branch so it will need to be resubmitted against master. Unfortunately, this pull request does not cleanly merge against the current master branch.

If you do nothing, this Pull Request will be automatically closed by @BrownTruck since it cannot be merged.

If this pull request is still valid, please rebase it against master (or merge master into it) and resubmit it against the master branch, closing and referencing the original Pull Request.

If you choose to rebase/merge and resubmit this Pull Request, here is an example message that you can copy and paste:

This introduces a new concept to pip called "command plugins". This allows a user to implement new features but do them outside pip itself. This means that people can do interesting experimental and/or custom features and implement these features outside of pip so that it doesn't clutter up the core code of pip and it doesn't require the time of pip maintainers to review it.

This might also be an alternative to simply ripping out features of pip that maybe aren't used a ton and are deemed not worth the effort to maintain them (e.g.: pip [un]zip, pip bundle, etc.). Perhaps even pip wheel could be made into a plugin (although I feel that it is probably used much more than the previously mentioned commands, but maybe since pip wheel requires the wheel package anyway, maybe it makes sense to just have the wheel package implement a pip plugin?). Such features could be removed from pip itself, but put into a separate plugin package and maintained by a 3rd party, so that they are not a time burden on PyPA developers.

To create a plugin command, simply:


Create a third-party Python distribution. It does not have to be on PyPI. It just needs to be installable.
Create a class that subclasses the pip.basecommand.Command class (example).
Register your new class with the new pip.command.v1 entry point in the setup.py of your distribution (example).


An example plugin is available at https:/msabramo/pipdeptree/tree/pip_plugin -- in this branch I've forked an already existing pipdeptree project that implements a separate pipdeptree command -- in my fork I've put in the necessary bits so that pipdeptree is a pip plugin.

Showing how it works...

[marca@marca-mac2 ~WORKON_HOME]$ mktmpenv
New python executable in tmp-4adc64ecf8c6ac9e/bin/python
Please make sure you remove any previous custom paths from your /Users/marca/.pydistutils.cfg file.
Installing setuptools, pip...done.
This is a temporary environment. It will be deleted when you run 'deactivate'.

[marca@marca-mac2 ~VIRTUAL_ENV]$ pip install https:/msabramo/pip/archive/command_plugins.zip
...
Successfully installed pip-6.1.0.dev0

# Install the pipdeptree plugin
[marca@marca-mac2 ~VIRTUAL_ENV]$ pip install https:/msabramo/pipdeptree/archive/pip_plugin.zip
Collecting https:/msabramo/pipdeptree/archive/pip_plugin.zip
  Downloading https:/msabramo/pipdeptree/archive/pip_plugin.zip
     - 20kB 85kB/s
Requirement already satisfied (use --upgrade to upgrade): pip>=1.4.1 in ./lib/python2.7/site-packages (from pipdeptree==0.4.2)
Installing collected packages: pipdeptree
  Running setup.py install for pipdeptree
    Installing pipdeptree script to /Users/marca/python/virtualenvs/tmp-4adc64ecf8c6ac9e/bin
Successfully installed pipdeptree-0.4.2

[marca@marca-mac2 ~VIRTUAL_ENV]$ pip

Usage:
  pip <command> [options]

Commands:
  ...
  deptree                     Display a dependency tree of the installed python packages

[marca@marca-mac2 ~VIRTUAL_ENV]$ pip deptree --help

Usage:
  pip deptree [options]

Deptree Options:
  -f, --freeze                Print names so as to write freeze files
  -a, --all                   list all deps at top level
  -l, --local-only            If in a virtualenv that has global access donot show globally installed packages
  -w, --nowarn                Inhibit warnings about possibly confusing packages
...

# Just installing a few packages so that the output of `pip deptree` is more interesting...
[marca@marca-mac2 ~VIRTUAL_ENV]$ pip install flask ipython
...

[marca@marca-mac2 ~VIRTUAL_ENV]$ pip deptree
wsgiref==0.1.2
Flask==0.10.1
  - Werkzeug [required: >=0.7, installed: 0.9.6]
  - Jinja2 [required: >=2.4, installed: 2.7.3]
    - MarkupSafe [installed: 0.23]
  - itsdangerous [required: >=0.21, installed: 0.24]
ipython==2.3.1
  - gnureadline [installed: 6.3.3]
wheel==0.24.0


Cc: @dstufft, @pfmoore, @naiquevin, @therealprologic

Refs: naiquevin/pipdeptree#1

---

*This was migrated from pypa/pip#2329 to reparent it to the ``master`` branch. Please see original pull request for any previous discussion.*

[BrownTruck]

This Pull Request was closed because it cannot be automatically reparented to the master branch and it appears to have bit rotted.

Please feel free to re-open it or re-submit it if it is still valid and you have rebased it onto master or merged master into it.

[szepeviktor]

@msabramo Are you planning to reopen it?

[msabramo]

@szepeviktor: Yep, see #4166.