Skip to content
This repository has been archived by the owner on Sep 9, 2020. It is now read-only.

internal/gps: ensure packages are deducible before attempting to solve #697

Conversation

ibrasho
Copy link
Collaborator

@ibrasho ibrasho commented Jun 1, 2017

An attempt to resolve #581.

I still think it could use some polishing, but I wanted to check if I'm moving in the right direction.

@ibrasho ibrasho force-pushed the ensure-packages-are-deducible-before-solving branch 2 times, most recently from bbde9bb to da33cb1 Compare June 1, 2017 01:25
Copy link
Member

@sdboyer sdboyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep! Definitely the right direction.

One thing, though. I realize I'm countermanding myself, but let's actually do this under solver.Solve() instead of Prepare(). These deductions have the possibility of triggering network activity, and I'd prefer to keep the work done in Prepare() strictly IO-free.

It'd be fine to add a solver.validateParams() method that does this, and call it nearly first thing, immediately before solver.selectRoot() in solver.Solve(). Chances are we'll have some more validation like this to add, and we can lump it into that method.

@ibrasho
Copy link
Collaborator Author

ibrasho commented Jun 1, 2017

@sdboyer Done. Note that Prepare() doc says the following:

// This function reads and validates the provided SolveParameters. If a problem
// with the inputs is detected, an error is returned.
Otherwise, a Solver is
// returned, ready to hash and check inputs or perform a solving run.

I've extracted the logic to a solver. validateParams() method. But I think it belongs to Prepare(), or that the comment should be clarified.

@ibrasho ibrasho force-pushed the ensure-packages-are-deducible-before-solving branch 2 times, most recently from c7a2389 to e159914 Compare June 1, 2017 19:55
@ibrasho
Copy link
Collaborator Author

ibrasho commented Jun 1, 2017

@sdboyer It seems that bridge.DeduceProjectRoot() allows race conditions.
solver.mtr is not protected against simultaneous writes by multiple goroutines, and it's not possible to get the gps.SourceManager directly to avoid using bridge.DeduceProjectRoot().

I tried to go back to Prepare(), but noticed that passing arbitrary packages to Prepare in tests fails.

Any preference on how to tackle this?
Adding a mutex over Solver.mtr in bridge.DeduceProjectRoot() will render goroutines useless since it will force sequential execution. I'm thinking of exposing the gps.SourceManager to the gps.solver.

@sdboyer
Copy link
Member

sdboyer commented Jun 2, 2017

@ibrasho ahh...crap.

Yes, the bridge allows races, because the entire solver is designed to be strictly single-threaded.

OK, given these issues, yes - let's move the verification back to Prepare(). I'm confused by that link you provide, though - I don't see any failures there?

@ibrasho
Copy link
Collaborator Author

ibrasho commented Jun 2, 2017

@sdboyer That test case adds 2 dependencies called baz and quz. And since they don't exist, Prepare() will return an error and the test will fail. In TestHashInputsOverrides, c is used multiple times too, causing failures.

This is what's causing CI builds to fail.

@sdboyer
Copy link
Member

sdboyer commented Jun 2, 2017

🤦‍♂️

sorry, of course, that make sense.

let me ponder a bit on the least invasive way to fix this.

@sdboyer
Copy link
Member

sdboyer commented Jun 2, 2017

OK, new plan. We don't actually need to do this validation within the solver, because it's all failures we'd arrive at eventually, anyway. That's a strong indication that this isn't actually "validation" in the sense that these conditions must be met for solver behavior to be defined. It's actually just preemptively doing work the solver would do later.

Let's do this validation as a standalone function in gps, and call that function directly from dep, prior to Solve()ing. That'll also give us more reusability, which we'll probably need to employ later within sourceGateway.getManifestAndLock(). We need to guard against this there, too - but in a separate PR.

Returning to the original panic encountered in #581, we can do yet another separate PR which is about gps' purely internal graceful handling of that case.

@ibrasho ibrasho force-pushed the ensure-packages-are-deducible-before-solving branch 2 times, most recently from b89b600 to 2058950 Compare June 3, 2017 17:25
@ibrasho
Copy link
Collaborator Author

ibrasho commented Jun 5, 2017

@sdboyer: This is passing now. But I still don't like it.

I'm creating another instance of gps.rootdata since it's not exported in the solver created by gps.Prepare().

Copy link
Member

@sdboyer sdboyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, i know i've been slow the last week - trying to catch up!

}

// ValidateParams validates the solver parameters to ensure solving can be completed.
func ValidateParams(params SolveParameters, sm SourceManager) error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need just a basic test for this - feed it some SolveParameters with both valid and invalid import paths.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are the tests in solver_test.go enough?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, yes, they are. but, how about we rename that file to solver_inputs_test.go, and move TestBadSolveOpts into there?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done.

@ibrasho ibrasho force-pushed the ensure-packages-are-deducible-before-solving branch from fc84b7b to f1d5b48 Compare June 15, 2017 20:29
Add gps.ValidateParams to ensure all packages in SolverParams are
deducible.

Signed-off-by: Ibrahim AshShohail <[email protected]>
@ibrasho ibrasho force-pushed the ensure-packages-are-deducible-before-solving branch from f1d5b48 to a961d76 Compare July 16, 2017 22:28
@sdboyer sdboyer closed this Jul 19, 2017
@sdboyer sdboyer reopened this Jul 19, 2017
@ibrasho ibrasho force-pushed the ensure-packages-are-deducible-before-solving branch 2 times, most recently from e8d34d0 to a961d76 Compare July 21, 2017 00:40
@sdboyer sdboyer reopened this Jul 26, 2017
@sdboyer
Copy link
Member

sdboyer commented Jul 26, 2017

i think the CLA issue should clear if a new commit comes in. maybe just rebase this and re-push, that should be enough.

@ibrasho ibrasho force-pushed the ensure-packages-are-deducible-before-solving branch from 1f42a17 to b6260d6 Compare July 27, 2017 17:26
@ibrasho
Copy link
Collaborator Author

ibrasho commented Jul 27, 2017

That trick actually works. 😁

Copy link
Member

@sdboyer sdboyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

little nit, looks like a debugging fmt.Println() survived

}

deducePkg := func(ip string, sm SourceManager) {
fmt.Println(ip)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like a debugging print?

@ibrasho ibrasho force-pushed the ensure-packages-are-deducible-before-solving branch from b6260d6 to 92df3dc Compare July 29, 2017 06:41
@ibrasho
Copy link
Collaborator Author

ibrasho commented Aug 4, 2017

Anything else needed here? 😁

Copy link
Member

@sdboyer sdboyer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nothing! in we go 😄

@sdboyer sdboyer merged commit 2e56bed into golang:master Aug 5, 2017
@sdboyer
Copy link
Member

sdboyer commented Aug 5, 2017

see, i said that, but then...yeah, we need this to be expanded, now that there are more places we actually solve/prepare params in ensure.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

panic: shouldn't be possible unable to deduce repository and source type for: "golang.org/x/sys/unix"
3 participants