Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFE: drop bundled distlib #10420

Closed
kloczek opened this issue Sep 1, 2021 · 14 comments
Closed

RFE: drop bundled distlib #10420

kloczek opened this issue Sep 1, 2021 · 14 comments
Labels
resolution: no action When the resolution is to not do anything type: feature request Request for a new feature

Comments

@kloczek
Copy link

kloczek commented Sep 1, 2021

Looks like pip and virtualenv ar eonly two remaining modules which are still somehow connected with distlib.
pip seems is only bundlig distlib.

[tkloczko@barrel SPECS]$ rpm -ql python-pip| grep distlib
/usr/lib/python3.8/site-packages/pip/_vendor/distlib
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__init__.py
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/__init__.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/__init__.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/__init__.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/compat.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/compat.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/compat.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/database.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/database.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/database.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/index.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/index.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/index.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/locators.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/locators.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/locators.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/manifest.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/manifest.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/manifest.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/markers.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/markers.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/markers.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/metadata.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/metadata.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/metadata.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/resources.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/resources.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/resources.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/scripts.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/scripts.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/scripts.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/util.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/util.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/util.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/version.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/version.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/version.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/wheel.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/wheel.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/__pycache__/wheel.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/__init__.py
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/__pycache__
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/__pycache__/__init__.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/__pycache__/__init__.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/__pycache__/__init__.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/__pycache__/misc.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/__pycache__/misc.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/__pycache__/misc.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/__pycache__/shutil.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/__pycache__/shutil.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/__pycache__/shutil.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/__pycache__/sysconfig.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/__pycache__/sysconfig.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/__pycache__/sysconfig.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/__pycache__/tarfile.cpython-38.opt-1.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/__pycache__/tarfile.cpython-38.opt-2.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/__pycache__/tarfile.cpython-38.pyc
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/misc.py
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/shutil.py
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/sysconfig.cfg
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/sysconfig.py
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/_backport/tarfile.py
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/compat.py
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/database.py
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/index.py
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/locators.py
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/manifest.py
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/markers.py
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/metadata.py
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/resources.py
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/scripts.py
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/t32.exe
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/t64.exe
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/util.py
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/version.py
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/w32.exe
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/w64.exe
/usr/lib/python3.8/site-packages/pip/_vendor/distlib/wheel.py

I was not able to find anything else which still may be using distlib.
Going as well to create similar ticket for virtualenv.

@DiddiLeija
Copy link
Member

I'm neutral on this. I don't know if doing this could have unintended side-effects.

@DiddiLeija DiddiLeija added the type: feature request Request for a new feature label Sep 2, 2021
@uranusjr
Copy link
Member

uranusjr commented Sep 2, 2021

Could you provide some context why you think depending on distlib is a bad thing?

@pfmoore
Copy link
Member

pfmoore commented Sep 2, 2021

Also, we use distlib to generate script wrappers, and there's (to my knowledge) no viable alternative to distlib. Any proposal to remove distlib would need to explain how we continue to provide that functionality. (Note - the hard case here is Windows, where executable wrappers are needed).

@kloczek
Copy link
Author

kloczek commented Sep 2, 2021

Could you provide some context why you think depending on distlib is a bad thing?

Simple I see that this module badly maintained.
For example test suite uses modules like compat, unittest2, cStringIO. I have problem even find them on pypi.

Also, we use distlib to generate script wrappers, and there's (to my knowledge) no viable alternative to distlib. Any proposal to remove distlib would need to explain how we continue to provide that functionality. (Note - the hard case here is Windows, where executable wrappers are needed).

I don't see anywhere else that this part of the 'distlib` is used.
Why not bundle that code where it is used?

@pfmoore
Copy link
Member

pfmoore commented Sep 2, 2021

Why not bundle that code where it is used?

Because we would then have to maintain it (possibly even including the C source code for the executable wrappers).

Let's just say this isn't going to happen. If someone were to release (and commit to maintaining) a replacement for the relevant parts of distlib, and someone submitted a PR to vendor the new library in place of distlib, then we might just possibly consider it. But even then, it's not obvious that such a PR would be accepted.

Simple I see that this module badly maintained.

Your time would probably be better spent in that case helping distlib to maintain their code (assuming they agree with you that it's "badly maintained" - you should probably be more respectful of the time the distlib maintainers have put into it, and start from an assumption that there are reasons why things are the way that they are, before campaigning to get projects to drop distlib...)

@ambv
Copy link
Contributor

ambv commented Sep 2, 2021

@kloczek, looks like distlib is still maintained: https://bitbucket.org/pypa/distlib/commits/

Last version released in May, with new commits after that. It uses some ancient backports and is still Python 2.7 compatible, this is not a huge problem per se.

@ambv
Copy link
Contributor

ambv commented Sep 2, 2021

OTOH distlib will be incompatible with Python 3.11 where we remove some deprecated undocumented functions from urllib.parse:
https://bitbucket.org/pypa/distlib/issues/152/distlib-is-incompatible-with-python-311

@kloczek
Copy link
Author

kloczek commented Sep 2, 2021

@ambv that is partially my point :P

Maintainer seems is taking more care about python 2.x (which already +1.5y EOSed) than keep in good shape for latest python :/
I'm packaging stuff and one of the requirements in which I'm trying to fit is include test suite execution during package process even on the cost of more modules/projects to be packaged.
As I wrote I have problems even with finding necessary modules to perform test suite execution :/
pip and virtualenv are modules which sits quite on top of whole dependencies pyramid. Only from that point of view cutting one module dependency more would be always some move towards make whole graph of the dependencies more planar (without loops).
I agree that on what I'm pointing is not-so-important .. nevertheless I've decided to probe such idea :)
I'm not 100% sure but looks like virtualenv uses only one function from that module .. if that would be cut off it will be possible to drop bundling distlib in pip as well.

@pradyunsg
Copy link
Member

FWIW, an alternative for generating script wrappers exists in installer now.

@pradyunsg
Copy link
Member

pradyunsg commented Sep 2, 2021

None the less, @kloczek's reasoning for filing this issue:

Simple I see that this module badly maintained.

This isn't based on facts, and isn't a good reason to drop distlib.

Given their history of making weird requests to Python projects (asking many poetry/flit-based projects to add a setup.py, among other things) -- they've filed 100s issues on across many projects -- I'm inclined to suggest that folks don't spend extended periods of time or energy on this issue.

@DiddiLeija
Copy link
Member

I think we should discuss this changes.

@DiddiLeija DiddiLeija added the state: needs discussion This needs some more discussion label Sep 2, 2021
@pfmoore
Copy link
Member

pfmoore commented Sep 2, 2021

I think we should discuss this changes.

Honestly, I don't think we sould unless someone comes up with an actual justification for the proposal (one that isn't simply unsubstantiated criticism of distlib).

@kloczek
Copy link
Author

kloczek commented Sep 2, 2021

Honestly, I don't think we sould unless someone comes up with an actual justification for the proposal (one that isn't simply unsubstantiated criticism of distlib).

I can point only on argument list of modules which are used by distlib test suite.
This says only that this module is more ofr less not tested.
Other is KISS principle.
And again .. nu rush.
My knowledge about some parts of some pip and virualenv code is not enough to point on some actiual advantages however I have kind of impression that therea are here some deposits of simplify few things (only because looks like that code not been touched quite long time). If that will be reached it will provide good return in longer term.

@ambv
Copy link
Contributor

ambv commented Sep 2, 2021

distlib is fine and this issue should be closed.

I reported a bug to them today about the pending Python 3.11 incompatibility and it's already fixed:

https://bitbucket.org/pypa/distlib/issues/152/distlib-is-incompatible-with-python-311

@pradyunsg pradyunsg added resolution: no action When the resolution is to not do anything and removed state: needs discussion This needs some more discussion labels Sep 2, 2021
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 3, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
resolution: no action When the resolution is to not do anything type: feature request Request for a new feature
Projects
None yet
Development

No branches or pull requests

6 participants