Rebuilding non-wanted locals

[snoyberg]

To see the problematic behavior, you can run the following in the wai repo (though other multi-package projects with inter-dependencies will work as well):

stack build --ghc-options=-O0 wai wai-extra
stack build --ghc-options=-O0 wai-extra

The first time, it will be wai, wai-logger, and wai-extra (as expected). The second call will result in all three packages being unregistered and rebuilt, even though it seems obvious that we only want to rebuild wai-extra. The reason is that in the second build, wai is no longer a wanted dependency, and therefore the extra GHC options do not apply to it. Because of that, the installed config options don't match the current config options, and therefore a reconfigure and rebuild is required.

I don't see an obvious better way to do things, unfortunately, without baking in special logic for different flags. The reason is this: consider the first build did not pass in -O0, and the second one did. We probably don't want to recompile wai just to disable optimizations.

One possibility is to mark some flags as "these don't make an installed package dirty", such as -O0. Even that is a bit problematic, since we now have some strange state following our build around such that stack build wai-extra will have different results depending on what sequence of commands was run previously.

I'm leaning towards saying that this is simply a corner case that stack will require a recompile for consistency, but I'm open to other ideas.

Pinging @nh2 who originally reported, and @mboes.

[nh2]

The reason is that in the second build, wai is no longer a wanted dependency, and therefore the extra GHC options do not apply to it. Because of that, the installed config options don't match the current config options, and therefore a reconfigure and rebuild is required.

For clarification, does this mean that when you stack build --ghc-options=-O0 wai-extra, it will reinstall wai, but reinstall it with -O?

[snoyberg]

Yes (or, more specifically, without any specific GHC options passed in, which is equivalent to -O).

[mboes]

I'm leaning towards saying that this is simply a corner case that stack
will require a recompile for consistency

Agreed. Worth an entry in the FAQ. The fix is to internalize the
ghc-options into the stack.yaml, if the user wants to avoid spurious
recompiles.

[snoyberg]

I've actually been playing with the idea of a --devel flag or something along those lines, which might be a good solution. Other possibilities are:

• Have a --ghc-options-all flag which even applies to non-wanted packages
• Do this via flags, which can already be targeted at specific packages

I think I like the idea of giving people an example in the FAQ of doing this with flags and leaving it at that. I don't want to turn the command line options into some deep, mystical incantation.

[nh2]

Yes

Hmm, this seems counter-intuitive to me. I would expect that if you stack install --ghc-options="..." somepackage, that all packages built by stack in this invocation really get these options passed.

[snoyberg]

I understand that being a logical conclusion, but it leads to bad places: you now have either:

• Implicit state in your build plan based on the flags used in previous builds, or
• Have to recompile every dependency being used each time you want to try a different set of flags on the current package

[nh2]

I see. So the fundamental problem is that when a user says stack install --ghc-options=... xyz, the user may want these options to apply to either of:

• just the xyz package
• the xyz package and all its dependencies
• the xyz package and those dependencies that stack decided would have to be built, independent of the ghc-options

The third point is the one I'd have expected, but it makes sense to me that you also want to support the first point (indeed that is useful).

The second point is hard to implement in full generality since that'd include base and you can't recompile that.

It seems to me that it's impossible to guess whether the user wants 1 or 3, and that an explicit choice would solve it.

[mboes]

Have to recompile every dependency being used each time you want to try a different set of flags on the current package

This doesn't so bad, if the upstream design choice can still be revisited. Arguably ghc-options are like cabal flags: it should be possible to specify applying them on a per-package basis. That way turning on or off ghc options globally incurs rebuilding everything, which is an intuitive behaviour, but if you just want to override the flags for some specific package then you can say so explicitly.

[snoyberg]

ghc-options-per-package is actually very logical, and fits with the philosophy I've been pushing for flags. I'd be concerned about it being too much of a user burden (which I think we're already hearing the per-package flag setting is, see #335). In other words: we've got the typical battle between control, concision, user friendliness, and robustness going on.

In theory, we could end up with three versions of ghc-options:

1. Apply to all packages (though I'd argue it shouldn't apply to snapshot packages, since we never tweak their build plans)
2. Apply to all wanted packages (current behavior)
3. Select a specific package to apply the options to

We could also add in the ability to specify ghc-options inside the stack.yaml file (which doesn't currently exist). Does that cover all things on the table?

[nh2]

Does that cover all things on the table?

I think so. To cover my use case, version 3 would work well in combination with a command that prints the packages that stack would install, so that I can do something like

stack install --ghc-options=... `stack install --dry-run --oneline mypackage`

Does that make sense?

Independent of this, it would be extremely helpful if stack could print the reason why something is unregistered (like GHC printing [TH] when it recompiles because of TH). That could maybe look like Unregistering wai [ghc-options changed].

[mboes]

I'd allow setting ghc-options for the snapshot packages too. I might want to say e.g. ghc-options: binary:-Ogazillion. Which would have the side effect of making binary a local package, no longer a snapshot package. But maybe you're including that in (3). Agreed about (1).

it would be extremely helpful if stack could print the reason why something is unregistered (like GHC printing [TH] when it recompiles because of TH). That could maybe look like Unregistering wai [ghc-options changed].

👍 --> separate ticket.

[mboes]

I'm mildly concerned that people will starting abusing the ghc-options field in the stack.yaml. Arguably build options should be self contained in the .cabal of each respective package. A stack.yaml ghc-options should only be used to override an upstream default.

[snoyberg]

To modify a snapshot package in any way, you need to first as it to extra-deps. I think this is good behavior to avoid confusion, and would like to keep it. Your point about ghc-options in stack.yaml makes sense. Let's not put them in.

So are we actually going to add the three different ways of doing GHC options on the command line? I'm still conflicted about it being too confusing.

[mboes]

To modify a snapshot package in any way, you need to first as it to extra-deps.

Except for snapshot packages that depend on a local package, right? But I guess if you want to modify the ghc-options of an implicitly pulled in snapshot package then asking that it be listed explicitly somewhere is fair enough.

So @nh2 swayed us both, but in the end I think I'm swinging back to your original position @snoyberg. Have flags like --no-optimizations and the like. These are "known clean", because they have a semantic meaning known to stack, contrary to --ghc-options=-O0. The latter may recompile more than you think, and that's okay, so long as it's documented in the FAQ.

The three varieties of ghc-options will be too confusing. And not solve the fundamental problem: that of avoiding previous invocations of stack from changing the output of following ones even if stack.yaml stays constant.

[snoyberg]

Except for snapshot packages that depend on a local package, right?

Doing that implicitly adds it to extra-deps. I suppose we could similarly implicitly add packages to extra-deps when they're mentioned in flags. If you like that idea, please open an issue.

I'm still not completely certain it's worth adding in the complexity around having some flags treated specially. Yes, there's a small overhead of packages being reconfigured and then rebuilt, but the rebuild should be fast due to reusing previous object files. Unless there's objection, I'd rather keep this as a doc-only fix.

Another aspect to this for me: the ide-backend and ghci support will likely end up providing even better means of doing quick testing, so I don't want to spend too much time prematurely optimizing on this side.

[mboes]

Unless there's objection, I'd rather keep this as a doc-only fix.

Yes, this is by now my position too as explained in my previous comment. @nh2 thoughts?

The recompiles could in theory be avoided by having stack build into a dir whose name is a function of --ghc-options. Doing so is a possible followup issue, but is it worth it?

[snoyberg]

I've updated the FAQ with a clarification about this, closing.