-
-
Notifications
You must be signed in to change notification settings - Fork 150
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: remove imports from the deprecated pip API #106
Conversation
Thanks for this. I'm away from work for a few days and have to take a proper look at this problem as well. Will update sometime next week. |
@Midnighter thanks for taking a stab at this, I can shed some light on how this is handled (I would love to see this land as we are using Firstly you have a few options, you can rely on In the import pkg_resources
dists = list(pkg_resources.working_set)
>>> print("[{dist.__class__!s}] : {dist!r}".format(dist=ws[0]))
[<class 'pkg_resources.DistInfoDistribution'>] : zope.deprecation 4.3.0 (/home/hawk/.pyenv/versions/3.7.0/lib/python3.7/site-packages)
>>> print("[{dist.__class__!s}] : {dist!r}".format(dist=ws[-1]))
[<class 'pkg_resources.EggInfoDistribution'>] : pythonfinder 1.1.0 (/home/hawk/git/pythonfinder/src) The local_only argument does two things:
To see if you are in a virtualenv, it's just: def running_under_virtualenv():
real_prefix = gettattr(sys, "real_prefix", None)
return True if real_prefix is not None else (sys.prefix != sys.base_prefix) This gets a bit tricky because Egg distributions don't actually work that way, so you need to expand your search path when looking for those specifically. Roughly speaking, you need to have the rules:
For simplicity this is the relevant code (roughly, not in any way tested): import site
import sys
from distutils import sysconfig as distutils_sysconfig
site_packages = distutils_sysconfig.get_python_lib()
try:
user_site = site.getusersitepackages()
except AttributeError:
user_site = site.USER_SITE
def find_egg(egg_dist):
search_locations = []
search_filename = "{0}.egg-link".format(egg_dist.project_name)
if running_under_virtualenv():
search_locations.append(site_packages)
if user_site:
search_locations.append(user_site)
else:
search_locations.append(site_packages)
if user_site:
search_locations.append(user_site)
search_locations.append(site_packages)
for site_directory in search_locations:
egg = os.path.join(site_directory, search_filename)
if os.path.isfile(egg):
return egg
def locate_dist(dist):
location = find_egg(dist)
if not location:
return dist.location That gives us a bunch of infrastructure to just add the following as our test function for local installations: def is_in_environment(dist):
if not running_under_virtualenv():
return True
return normalize(locate_dist(dist)).startswith(normalize(sys.prefix)) The def is_in_usersite(dist):
return normalize(locate_dist(dist)).startswith(normalize(user_site)) And we can add a final summarizing function as there was before: def dummy_test(dist):
return True
def get_installed_distributions(user_only=False, local_only=True):
test_local = is_in_environment if local_only else dummy_test
test_user_site = is_in_usersite if user_only else dummy_test
return [
dist for dist in pkg_resources.working_set
if test_local(dist) and test_user_site(dist)
] Putting that all together: import pkg_resources
import os
import site
import sys
from distutils import sysconfig as distutils_sysconfig
site_packages = distutils_sysconfig.get_python_lib()
try:
user_site = site.getusersitepackages()
except AttributeError:
user_site = site.USER_SITE
def running_under_virtualenv():
real_prefix = gettattr(sys, "real_prefix", None)
return True if real_prefix is not None else (sys.prefix != sys.base_prefix)
def normalize(path):
return os.path.normcase(os.path.abspath(os.path.expanduser(path)))
def find_egg(egg_dist):
search_locations = []
search_filename = "{0}.egg-link".format(egg_dist.project_name)
if running_under_virtualenv():
search_locations.append(site_packages)
if user_site:
search_locations.append(user_site)
else:
search_locations.append(site_packages)
if user_site:
search_locations.append(user_site)
search_locations.append(site_packages)
for site_directory in search_locations:
egg = os.path.join(site_directory, search_filename)
if os.path.isfile(egg):
return egg
def locate_dist(dist):
location = find_egg(dist)
if not location:
return dist.location
def is_in_environment(dist):
if not running_under_virtualenv():
return True
return normalize(locate_dist(dist)).startswith(normalize(sys.prefix))
def is_in_usersite(dist):
return normalize(locate_dist(dist)).startswith(normalize(user_site))
def dummy_test(dist):
return True
def get_installed_distributions(user_only=False, local_only=True):
test_local = is_in_environment if local_only else dummy_test
test_user_site = is_in_usersite if user_only else dummy_test
return [
dist for dist in pkg_resources.working_set
if test_local(dist) and test_user_site(dist)
] |
For now I have quick-fixed the ImportError by updating the import statement as per the changes pip version 18.1. I would be interested in a long term fix such as this one. But not being a pipenv user I need to catch up with the developments happening there. Will have a look at pip-shims, I quite liked the idea upon a quick glance at it's README. |
@naiquevin the fix outlined above is kind of independent of pipenv, it mostly relates to the fact that pip is maintaining an internal API that they are likely to continue breaking going forward. I already have fixes for all of the broken things merged into pipenv's master branch -- this was the change we used for the next release: https:/pypa/pipenv/blob/master/tasks/vendoring/patches/vendor/pipdeptree-updated-pip18.patch |
Interesting approach. How do you make sure a shim is added when internal api changes? Or is it something that can only be done after some one reports a problem when a new version of pip is released? |
https://travis-ci.com/sarugaku/pip-shims/builds/87356090 => it's now on a nightly cron to build against the master branch of pip, so ideally I'd have some warning when things break |
4d06074
to
3c2dbce
Compare
hi, with pip-21.3 coming in 2 weeks , so a solution, maybe this patch, is needed to keep pipdeptree alive |
As a note, I've moved the functionality that I care about to https:/Midnighter/dependency-info. It's based entirely on importlib metadata. It only has a very minimal feature set, though. |
Closing as this seems to have stalled. |
I think this should be a good start to move away entirely from pip. The only part I'm not clear about is the
local_only
anduser_only
flag. I'm not sure how to influence that for thepkg_resources.working_set
.