Add --fail-fast

[tgdwyer]

Added a command line option "--stop-on-fail".
When in --stop-on-fail mode, after a test in an example group fails, running tests from subsequent example groups is aborted.

[sol]

Can you please try the following: Add a test case with a failing $setup and run it with --fail-fast, then observe if it will abort early. My suspicion is no, it won't.

To address this you may need to add shortcut semantics to the stack (e.g. add MaybeT to the Report type).

Ideally this will allow you to handle early abortion in reportFailure / reportError.

[sol]

Regarding the failing CI job, you'll need to run hpack to update the .cabal-file.

[sol]

@tgdwyer just to be sure, because the tests take a long time to run, you can focus on individual test cases by changing it to fit (or alternatively focus on a group of test cases by changing describe to fdescribe. This is usually more convenient then passing --match on the command line.

https://hackage.haskell.org/package/hspec-2.11.9/docs/Test-Hspec.html#g:5

[tgdwyer]

I have renamed the option to --fail-fast and made the various merges and cleanups.

I did some experimenting with $setup and it looks like it fast fails already. For the following test:

module Foo where

-- $setup
-- >>> let x = 1 + "hello"

-- | foo
-- >>> foo
-- 42
foo :: Int
foo = 42

doctest (whether with --fast-fail or not) produces:

test/extract/setup/Foo.hs:4: failure in expression `let x = 1 + "hello"'
expected:
but got:
^
:27:9: error:
• No instance for (Num String) arising from the literal ‘1’
...
Examples: 2 Tried: 1 Errors: 0 Failures: 1

Meanwhile, I agree it would probably better to use MaybeT to respond to fast fail than just looking for failures in the Report counts. I would like to investigate this, but won't have time for a few weeks. Any such refactoring would probably need to be part of a separate PR.

Anything else?

[sol]

I did some experimenting with $setup and it looks like it fast fails already.

I think what's happening here is that, if $setup fails, all examples from the current module will be skipped. My suspicion is that it would still continue with other modules. So to reproduce the "issue" you would need a test case with multiple modules.

I somehow missed that on my first review.

[tgdwyer]

My suspicion is that it would still continue with other modules. So to reproduce the "issue" you would need a test case with multiple modules.

I've changed the logic of runModule to skip if failures > 0 (rather than only on increasing failure count). Thus, after the first failure all subsequent examples will be skipped regardless of module.
I've added another test case with a second file to confirm its module is skipped.

I've also run hpack so the test files are included in doctest.cabal and the CI should run.

[tgdwyer]

I think I've implemented all the requested changes and that it's ready for merge, if you're happy?

[sol]

I think this is still not working correctly if you have multiple $setup blocks (see my other comment).

Would you mind cherry-picking bbe00c6 and using this as a starting point?

[sol]

@tgdwyer also cherry-pick b4a777c, or just rebase on https:/sol/doctest/tree/maybe-t.

[tgdwyer]

@tgdwyer also cherry-pick b4a777c, or just rebase on https:/sol/doctest/tree/maybe-t.

Done! Thanks for wrapping Report in MaybeT! FailFast logic is much better now. Latest commits just use forM_ to traverse the Maybe effect in runModule, same as runModules. Code is much cleaner.

[sol]

Great, the code looks good now 👍

Let's clean up the test descriptions, and then it's good to go.

In addition, I think you need to run hpack again.