[POC] Improve importlib.metadata
usage
#854
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Roughly a POC for now that will most likely get split into multiple PRs in the end, but thought it would be nice to have a first draft, to raise some discussion.
While working towards how to resolve #827, I dug a bit into
importlib.metadata
that we currently use, to see if we have some ways to improve what we do.Python 3.10 introduced
packages_distributions
that returns a mapping that maps Python module names to the packages that expose them, by reading fromtop_level.txt
like we do. In Python 3.11, the method became even smarter by falling back to reading fromRECORD
when notop_level.txt
is present, also exactly like we do.Updates in
importlib.metadata
are upstreamed fromimportlib_metadata
, so by requiring>=4.13
on Python < 3.11, we could usepackages_distributions
in our codebase for 2 things.First, avoid parsing
top_level.txt
andRECORD
ourselves, and instead simply rely on the method which already does that.Second, and this is the most interesting part, detect more transitive dependencies, as I noticed that when building
Module
, we also rely onimportlib.metadata
and check if we detect a package <-> module mapping, but by usingmetadata
fromimportlib.metadata
, which is AFAIK equivalent to only checkingtop_level.txt
, and notRECORD
, meaning that some dependencies are mistakenly flagged as missing instead of transitive dependencies (like shown in #827).With the changes in this PR,
bs4
import in the issue is correctly reported as a transitive dependency, instead of a missing one:Switching to
packages_distributions
also highlights an issue with our implementation. InModule
, we assume that we can only have one package exposing a module, but in fact, it is possible for multiple packages to provide a module (for instance if using namespace packages I believe). Because of that, the report we do might be incomplete, but worse than that, this probably means that we wrongly handle some violations today, as for instance, if a module is provided by 2 packages, where one is a dev dependency and one a production one, we would need to assess the violations based on all packages we find.As mentioned at the top of the description, I'd like to eventually split the PR into smaller pieces:
importlib_metadata
on Python < 3.11 and updateDependency
to usepackages_distributions
, removing our own logic that reads fromtop_level.txt
andRECORD
along the wayModule
to usepackages_distributions
, which is more complex to handle since we could now end up with multiple packages when building a module