2020 resolver: get_topological_weights AssertionError #9031
Comments
jahn
changed the title
pradyunsg
added
C: new resolver
kind: crash
|
This is causing my rtd builds to fail (https://readthedocs.org/projects/dclab/builds/12162768/). My workaround was to pin sphinx to |
|
Thanks @jahn for a detailed report here! It has been super helpful to have the reproducer as well as useful debugging information for this issue. Visualising the graph here, it looks like we're getting a malformed graph from resolvelib. Graph on 20.2.3 (good):
Graph on 20.2.4 (naughty):
Notice how the node for "sphinxcontrib-websupport" has sneaked into this one? That's causing the failure here. A quick dump of words to describe things:
It seems to me that this issue arises because that we're adding nodes that have no "path to root" in the graph, by adding the parent unconditionally. IOW, this is a bug in resolvelib's SOOOO. Possible choices to make here:
Things to do here:
# This was the blob of code that I used to generate the input to graphviz, for those graphs.
print('digraph G {')
print(' rankdir=LR;')
for node, others in graph._forwards.items():
print(f' "{node}";')
for node, others in graph._forwards.items():
for to_node in others:
print(f' "{node}" -> "{to_node}";')
print('}')/cc @pfmoore because I couldn't find a smooth excuse to mention you earlier than this line. :) |
|
I figured that we're only seeing the installed version of And, yes, that works. Yay! Now, it's almost 4am. I'mma sleep. |
|
Nice analysis! I think dropping It would be nice to not have that node in the graph, but I don't feel that's a critical issue as we know it's not a genuine issue with the resolution or ordering. I'm not 100% clear whether that's possible solely in resolvelib or whether it would need some help from pip, but that's something we can work out. Maybe change the assertion to a warning, so we don't completely lose the information that something odd is going on. And maybe if we can create a test to demonstrate the issue (make it xfail until we deal with it) that would also help us not to forget about it. Brain dump over. Oh, and @jahn can I add my thanks for a very well written, detailed and helpful issue report. It made it much easier to understand what's going on here. |
This is definitely solvable purely on the resolvelib side -- we'd basically need to rework the graph construction slightly to do the right thing. |
|
Can you check whether assert len(weights) == len(result.mapping) + 1 # Plus the santinel None node.instead. I will find some time to also fix it on resolvelib’s side as well, assuming the |
|
Darn, I’m having a hard time trying to come up with an simplified example. It seems that a simple backtrack is not enough to cause the issue. This is what I was trying to use as a test case: Now I instruct the resolver to resolve coffee before tea. This would induce backtracking because Not sure what I’m missing 😞 |
|
I thought a key point with the original example was that |
Signed-off-by: Keith Rothman <[email protected]>
Signed-off-by: Keith Rothman <[email protected]>
|
I persumed ordering is key as well, which is why I made the newer version of I also tried mimicking the already installed behaviour by swapping and instruct the provider to put Here’s the test code I’m using, maybe it’s me implementing things wrong… def test_result_cleanup():
dependencies = {
("kettle", 0): [],
("kettle", 1): [("stove", {})],
("water", 0): [],
("tea", 0): [("kettle", {0})],
("coffee", 0): [("water", {0}), ("kettle", {1})],
("coffee", 1): [("kettle", {0})],
}
pref_vers = {"coffee": {0: 1}} # Prefer coffee-0 over coffee-1.
class Provider(AbstractProvider):
def identify(self, dependency):
return dependency[0]
def get_preference(self, resolution, candidates, information):
# Resolve coffee before tea. This will cause backtracking.
if candidates[0][0] == "tea":
return 1
return 0
def get_dependencies(self, candidate):
return dependencies[candidate]
def find_matches(self, requirements):
if not requirements:
return []
name, versions = requirements.pop()
versions = {
version
for version in versions
if all(version in r[1] for r in requirements)
}
# Prefer coffee-0 over coffee-1.
if name == "coffee":
return sorted((name, v) for v in versions)
return sorted(((name, v) for v in versions), reverse=True)
def is_satisfied_by(self, requirement, candidate):
return candidate[1] in requirement[1]
resolver = Resolver(Provider(), BaseReporter())
result = resolver.resolve([("coffee", {0, 1}), ("tea", {0})])
assert "water" in result.graph, list(result.graph)
# AssertionError: [None, 'kettle', 'tea', 'coffee'] |
|
Ah I see, sorry. But in the actual example, |
Fixup doc requirements to work around pypa/pip#9031
Yup -- this works. :) |
Get around the pip problem by removing sphinx explicitly: See the issue here: pypa/pip#9031 Sphinx will still be installed because the rtd-theme depends on it.
|
I just stumbled on this error; I see it's fixed already in 20.3 (and could confirm that it works for my use case in 20.3b1) - but while waiting for 20.3 to drop, what can I do to figure out what packages I need to lock/remove in my project as a workaround? |
|
There is a 20.3b1 you can install. |
|
The package that is already installed in the environment that would be updated. For ReadTheDocs builds, that's gonna be Sphinx. |
|
@uranusjr Yes - I know (as stated, I've already tested that my use case works on 20.3b1); my point was that I might not be able to do this in the production environment, so I would like to know what I could do with my deps while waiting for a non-beta version. |
pypa/pip#9031 is fixed so we can include sphinx in requirements.txt Fixed Makefile so that it will use a virtual environment if one is active. Added some simple directions for building docs.
pypa/pip#9031 is fixed so we can include sphinx in requirements.txt Fixed Makefile so that it will use a virtual environment if one is active. Added some simple directions for building docs.
pypa/pip#9031 is fixed so we can include sphinx in requirements.txt Fixed Makefile so that it will use a virtual environment if one is active. Added some simple directions for building docs.
What did you want to do?
After installing sphinx 1.8.5, pip freeze gives this:
Output
Additional information
Dependency tree (before error):
When installing sphinx and sphixcontrib-bibtex, sphinx is set to upgrade to 3.2.1. The error only appears when sphinx is given on the command line. When only sphinxcontrib-bibtex is installed, it works fine and sphinx is upgraded. The error appears both in 20.2.4 and master.
This is the value of
weightsin get_topological_weights:This is
graph:This is
graph._vertices:sphinxcontrib-websupport is missing from weights.
This is
graph._forwards:This is
graph._backwards:The text was updated successfully, but these errors were encountered: