Isolate, reuse PackageFinder best candidate logic #5971

uranusjr · 2018-10-30T21:12:39Z

My take on #5175. This turns out to be more complicated than I imagined. I decided to make the outdated check follow how pip finds versions for requirements passed by the user:

If allow_all_prereleases is True, consider prereleases. (not applicable for self version check)
Try to find the latest stable version.
If no stable versions are available, try to offer prereleases. (PEP 440)

I also tweaked the code in find_requirement a little to refactor out some duplicate formatting.

Fix #5175, close #5928.

cjerdonek · 2018-10-31T08:53:24Z

Question / observation: to make this much more easily testable (and easier to understand, etc), it looks like you might be able to convert both of find_requirement() and find_best_match() into pure functions that accept a return value of find_all_candidates() (and some other simpler arguments). Then find_requirement() could be implemented as calling find_all_candidates(), followed by the one-line call to the pure function.

Does that look right to you?

uranusjr · 2018-11-01T06:01:17Z

I considered that approach as well, but opted for the least-effort one in the end. I’ll try it instead. The call in pip_version_check would look a little awkward, but that is probably because of the convoluted state of PackageFinder. Hopefully I can untangle things more with future refactoring…

cjerdonek · 2018-11-08T11:19:25Z

Coming back to this (I was busy with other things lately), I think your previous version might have been better. I don't know for sure because I can't see it any longer, but it might be because of the awkwardness you mentioned -- without a similar gain to offset it. My suggestion of passing the return value of find_all_candidates() was only if it would result in being able to factor out a pure function (which would be the gain). That's what I was asking about the possibility of. Looking at the code more closely myself after seeing your latest version, it looks like PackageFinder._candidate_sort_key() is preventing this from being done easily. Maybe you encountered that yourself when you tried.

uranusjr · 2018-11-09T04:53:16Z

Yes, indeed _candidate_sort_key is the most problematic part. I have the previous version locally, but looking at it now, I feel both versions have some better parts. I’ll need to spend sometime (maybe this weekend) merging the good parts together.

uranusjr · 2018-11-09T07:28:19Z

I’ve worked on this a bit, and have a sort of design problem in refactoring. An easy way to implement this would be to expose three methods:

find_all_candidates(name) finds all candidates
filter_applicable_candidates(candidates, specifier) takes the complete list of candidates, and return only ones matching the specifier.
select_best_candidate(candidates) choose the best candidate (with _candidate_sort_key) from the applicable candidates.

Now both find_requirement() and pip_version_check() can call these three methods to do what they want. But this is a very bad interface, and invites users to implement things incorrectly. What if someone forgets to call filter_applicable_candidates()? It would then select from the wrong candidate list, but it is extremely unlikely to be caught, and ever more difficult to debug. The better interface here is to expose only two methods:

find_all_candidates(name) (same)
find_best_candidate(name, specifier) is the combination of the three previous methods

If you want all candidates, use the first one. If you only want to know what to install, use the second one. No way to go wrong. The problem is, however, find_requirement() relies on intermediate values (all_candidates and applicable_candidates) to do logging, and the combined interface does not allow that. (In the first version, I returned a three-tuple to contain all information, but I think that is terrible.)

The current version is what I consider a middle ground. The three methods are kept, but marked as internal (except find_all_candidates; it is already public and has users). The call sequence is implemented in find_best_candidate as public interface, but find_requirement does not use that, instead call the individual functions to get the intermediate values it wants.

I don’t think this is a good implementation, but is the correct design. If this can be merged, I will submit another purely refactoring PR to deal with the duplicate code, probably by introducing an internal structure to hold the three internal values.

src/pip/_internal/index.py

pradyunsg · 2018-11-09T07:44:28Z

Looking at the current version, I think it does what's needed well enough; and @uranusjr's suggested plan moving forward sounds fair to me.

I'll defer to @cjerdonek since he's been more involved here than I have. :)

uranusjr · 2019-03-30T09:45:34Z

Soft ping :)

cjerdonek

Thanks for pinging / reviving this. I appreciate your work and thoughtful approaches.

Some comments.

news/5175.bugfix

cjerdonek · 2019-03-30T13:21:44Z

src/pip/_internal/index.py

 if applicable_candidates:
- best_candidate = max(applicable_candidates,
- key=self._candidate_sort_key)
+ best_candidate = self._select_best_candidate(applicable_candidates)


I read and appreciate your comments describing your thinking on this. Thanks. While I think I understand why you did it, on balance it seems to me like it would be better to avoid copying the exact three lines right next to each other, even if this is something you plan to fix later. It seems like it would be simpler and clearer and less error-prone to have them both go through the same function.

Two easily implementable alternatives, both of which I'm sure you've already thought of, and either of which I'd be okay with--

You could add a return_intermediate_values=False argument that find_requirement() could pass True for. Passing True would cause the return value to be a tuple rather than a single value. It's ugly, but would get the job done (and at least it wouldn't affect pip_version_check()'s invocation).

Kind of like subprocess.run()'s API that returns a CompletedProcess object, you could return an object instead of a single value. You could even have two variants: analyze / plan / calculate_best_candidate() that returns an object containing auxiliary info, and a simpler wrapper function find_best_candidate() that pip_version_check() could use that returns just the final value. As for (1), the latter would make it so pip_version_check() would be unaffected by the extra info.

The CompletedProcess model sounds like a good idea :D In retrospect it is not unlike the original tuple solution, but with a better interface.

cjerdonek · 2019-03-30T13:27:34Z

src/pip/_internal/utils/outdated.py

- all_candidates = finder.find_all_candidates("pip")
- if not all_candidates:
+ try:
+  candidate = finder.find_best_candidate("pip", SpecifierSet())


It seems like the caller shouldn't have to know about SpecifierSet objects to call find_best_candidate(). What about making that argument optional?

I am more fond of being explicit :) Convenience is not as important here since the API is internal (anyone wanting to understand the this line need to know about specifiers anyway), and always requiring a SpecifierSet reduces the chance of forgetting to pass one when you should.

Would finder.find_best_candidate("pip", specifier=None) be an acceptable middle ground?

I concur omitting the specifier should be okay, since finder.find_best_candidate("pip") is clear enough in terms of communicating what's happening.

I'm okay with all three options on the table, only slightly preferring the form I've stated here.

src/pip/_internal/utils/outdated.py

pypa-bot · 2019-04-02T06:56:51Z

Hello!

I am an automated bot and I have noticed that this pull request is not currently able to be merged. If you are able to either merge the master branch into this pull request or rebase this pull request against master then it will eligible for code review and hopefully merging!

Split out how PackageFinder finds the best candidate, and reuse it in the self version check, to avoid the latter duplicating (and incorrectly implementing) the same logic.

uranusjr · 2019-04-02T07:02:41Z

This looks even cleaner than I expected! Thanks so much @cjerdonek 😀

(Also rebased to keep the bot happy.)

cjerdonek · 2019-04-03T09:29:37Z

I have some comments after the latest changes.

cjerdonek

Lots of changes in this re-write of the PR, so some new comments.

src/pip/_internal/utils/outdated.py

src/pip/_internal/index.py

cjerdonek · 2019-04-03T09:56:42Z

src/pip/_internal/index.py

 if best_installed:
 # We have an existing version, and its the best version
 logger.debug(
 'Installed version (%s) is most up-to-date (past versions: '
 '%s)',
 installed_version,
- ', '.join(sorted(compatible_versions, key=parse_version)) or


It looks like parse_version got dropped in the new code.

This is merged into the above variable. I have made this into a function instead, with additional comments to make the usage clearer (hopefully).

src/pip/_internal/index.py

cjerdonek · 2019-04-03T10:08:54Z

src/pip/_internal/index.py

+ def __init__(
+ self,
+ candidates, # type: List[InstallationCandidate]
+ specifier, # type: specifiers.BaseSpecifier


Again, I think specifier should be optional (defaulting to the empty SpecifierSet).

I made the specifier argument optional in finder.find_candidates(), but kept it required in FoundCandidates constructor.

Also fix the missing parse_version call, and add comment for it.

uranusjr · 2019-04-16T11:04:47Z

Are there things I could do to push this forward? Should I change FoundCandidates’s specifier init argument to optional? I feel it is quite unnecessary since the class is only intended to be instantiated by PackageFinder, and default values only makes things less obvious.

cjerdonek · 2019-04-16T11:11:00Z

Hi @uranusjr, sorry! I actually just spent time going over this again an hour or so ago. I had been wanting to go over it again for a while but only just now got to it. I had some further minor thoughts, but I think it looks good enough. I was planning to merge it first thing tomorrow. Thanks for your thought and work on this and for your patience.

cjerdonek · 2019-04-16T11:17:52Z

PS - if there were further tweaks you wanted to do, you could always do that in a later PR. (And if it's a smaller / easier PR, I promise reviews can happen way faster.) It's only when there's more going on in a PR that it slows things down.

cjerdonek · 2019-04-16T11:37:38Z

Regarding FoundCandidates's init method, I was actually realizing it would be simpler to pass the set of versions to filter by, instead of passing in specifier and prereleases and recomputing that set each time. For example, since the set is needed anyways, it seems like it shouldn't have to be computed more than once (including even just for formatting). But it would be better to discuss further improvements after merging what you already have IMO, as opposed to postponing further.

uranusjr · 2019-04-16T13:16:18Z

Ah, I see what you mean, good point. I’ll work on improving that after this gets merged, then.

cjerdonek · 2019-04-16T23:48:15Z

Thanks again for your work on this, @uranusjr! 👍Can you double-check that the issues this closes are closed?

cjerdonek · 2019-04-16T23:49:01Z

src/pip/_internal/index.py

+ def _format_versions(cand_iter):
+ # This repeated parse_version and str() conversion is needed to
+ # handle different vendoring sources from pip and pkg_resources.
+ # If we stop using the pkg_resources provided specifier.


PS - also, here's a code comment that got chopped off that can be fixed in a subsequent commit.

uranusjr · 2019-04-16T23:50:55Z

I believe all related issues/PRs (at least those mentioned) are closed 🍿

lock · 2019-05-28T05:07:11Z

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

uranusjr force-pushed the pip-latest-version-prerelease branch 2 times, most recently from e5bfbdf to df68f3b Compare October 30, 2018 23:59

cjerdonek added T: bugfix C: finder PackageFinder and index related code labels Oct 31, 2018

uranusjr force-pushed the pip-latest-version-prerelease branch from df68f3b to 9b20265 Compare November 1, 2018 07:06

uranusjr force-pushed the pip-latest-version-prerelease branch 2 times, most recently from 1b7026f to 0f9f23a Compare November 9, 2018 07:10

pradyunsg reviewed Nov 9, 2018

View reviewed changes

src/pip/_internal/index.py Outdated Show resolved Hide resolved

uranusjr mentioned this pull request Mar 30, 2019

Select file to download based on existing hashes when --require-hashes #3634

Closed

cjerdonek reviewed Mar 30, 2019

View reviewed changes

uranusjr force-pushed the pip-latest-version-prerelease branch from 65cae6e to 1e85cf7 Compare April 2, 2019 06:56

pypa-bot added the needs rebase or merge PR has conflicts with current master label Apr 2, 2019

Isolate, reuse PackageFinder best candidate logic

ea1d5ac

Split out how PackageFinder finds the best candidate, and reuse it in the self version check, to avoid the latter duplicating (and incorrectly implementing) the same logic.

uranusjr force-pushed the pip-latest-version-prerelease branch from 1e85cf7 to ea1d5ac Compare April 2, 2019 07:00

pypa-bot removed the needs rebase or merge PR has conflicts with current master label Apr 2, 2019

pradyunsg approved these changes Apr 3, 2019

View reviewed changes

cjerdonek reviewed Apr 3, 2019

View reviewed changes

uranusjr added 3 commits April 8, 2019 15:39

Turn FoundCandidates properties into functions

1e3b4db

Also fix the missing parse_version call, and add comment for it.

Make specifier argument optional

adc498a

Keep linters happy

4dfda6b

Fix mock

2882042

uranusjr force-pushed the pip-latest-version-prerelease branch from ce8a021 to 2882042 Compare April 8, 2019 08:27

cjerdonek merged commit 14cb4f4 into pypa:master Apr 16, 2019

cjerdonek reviewed Apr 16, 2019

View reviewed changes

uranusjr deleted the pip-latest-version-prerelease branch April 16, 2019 23:50

uranusjr mentioned this pull request Apr 17, 2019

Finder found candidates cleanup #6415

Merged

cjerdonek mentioned this pull request Apr 17, 2019

Ignore pre-release versions when prompting for upgrade #5470

Closed

lock bot added the auto-locked Outdated issues that have been locked by automation label May 28, 2019

lock bot locked as resolved and limited conversation to collaborators May 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Isolate, reuse PackageFinder best candidate logic #5971

Isolate, reuse PackageFinder best candidate logic #5971

uranusjr commented Oct 30, 2018 •

edited

Loading

cjerdonek commented Oct 31, 2018

uranusjr commented Nov 1, 2018

cjerdonek commented Nov 8, 2018

uranusjr commented Nov 9, 2018

uranusjr commented Nov 9, 2018

pradyunsg commented Nov 9, 2018

uranusjr commented Mar 30, 2019

cjerdonek left a comment

cjerdonek Mar 30, 2019

uranusjr Apr 2, 2019

cjerdonek Mar 30, 2019

uranusjr Apr 2, 2019

pradyunsg Apr 3, 2019

pypa-bot commented Apr 2, 2019

uranusjr commented Apr 2, 2019

cjerdonek commented Apr 3, 2019

cjerdonek left a comment

cjerdonek Apr 3, 2019

uranusjr Apr 8, 2019

cjerdonek Apr 3, 2019

uranusjr Apr 8, 2019

uranusjr commented Apr 16, 2019

cjerdonek commented Apr 16, 2019

cjerdonek commented Apr 16, 2019

cjerdonek commented Apr 16, 2019

uranusjr commented Apr 16, 2019

cjerdonek commented Apr 16, 2019

cjerdonek Apr 16, 2019

uranusjr commented Apr 16, 2019

lock bot commented May 28, 2019

Isolate, reuse PackageFinder best candidate logic #5971

Isolate, reuse PackageFinder best candidate logic #5971

Conversation

uranusjr commented Oct 30, 2018 • edited Loading

cjerdonek commented Oct 31, 2018

uranusjr commented Nov 1, 2018

cjerdonek commented Nov 8, 2018

uranusjr commented Nov 9, 2018

uranusjr commented Nov 9, 2018

pradyunsg commented Nov 9, 2018

uranusjr commented Mar 30, 2019

cjerdonek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pypa-bot commented Apr 2, 2019

uranusjr commented Apr 2, 2019

cjerdonek commented Apr 3, 2019

cjerdonek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uranusjr commented Apr 16, 2019

cjerdonek commented Apr 16, 2019

cjerdonek commented Apr 16, 2019

cjerdonek commented Apr 16, 2019

uranusjr commented Apr 16, 2019

cjerdonek commented Apr 16, 2019

Choose a reason for hiding this comment

uranusjr commented Apr 16, 2019

lock bot commented May 28, 2019

uranusjr commented Oct 30, 2018 •

edited

Loading