Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#19 -- Record information on multiple expectation failure an include in report #20

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

gregmuellegger
Copy link

@gregmuellegger gregmuellegger commented Feb 22, 2024

This is a proposed fix for #19

It records all failing expectations and prints those during reporting.

Here is a working example:

devtools::install_github("gregmuellegger/r-hedgehog", ref = "19-fix-expect-failure-reporting")

library(testthat)
library(hedgehog)

test_that("Demo: shows all failing expectations now", {
    forall(
        gen.element(1:10),
        function (n) {
            expect_equal(n, n + 1)
            expect_equal(n, n + 2)
            
            expect_equal(n, n)
        }
    )
})

output:

── Failure: Demo: shows all failing expectations now ───────────────────────────
Falsifiable after 1 tests, and 1 shrinks

<expectation_failure/expectation/error/condition>
`n` not equal to n + 1.
1/1 mismatches
[1] 1 - 2 == -1
Backtrace:
     ▆
  1. └─hedgehog::forall(...)
  2.   └─hedgehog:::run.prop(property, counterexample$smallest, curry)
  3.     ├─base::tryCatch(...)
  4.     │ └─base (local) tryCatchList(expr, classes, parentenv, handlers)
  5.     │   └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
  6.     │     └─base (local) doTryCatch(return(expr), name, parentenv, handler)
  7.     ├─base::withCallingHandlers(...)
  8.     ├─base::do.call(property, arguments)
  9.     └─`<fn>`(1L)
 10.       └─testthat::expect_equal(n, n + 1)

<expectation_failure/expectation/error/condition>
`n` not equal to n + 2.
1/1 mismatches
[1] 1 - 3 == -2
Backtrace:
     ▆
  1. └─hedgehog::forall(...)
  2.   └─hedgehog:::run.prop(property, counterexample$smallest, curry)
  3.     ├─base::tryCatch(...)
  4.     │ └─base (local) tryCatchList(expr, classes, parentenv, handlers)
  5.     │   └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
  6.     │     └─base (local) doTryCatch(return(expr), name, parentenv, handler)
  7.     ├─base::withCallingHandlers(...)
  8.     ├─base::do.call(property, arguments)
  9.     └─`<fn>`(1L)
 10.       └─testthat::expect_equal(n, n + 2)

Counterexample:
[1] 1

@gregmuellegger gregmuellegger changed the title WIP: #19 -- Only record information on expectation failure #19 -- Record information on multiple expectation failure an include in report Feb 22, 2024
@HuwCampbell
Copy link
Member

I'll have a good look at this as soon as I get chance.

Looks mostly right, but the output is a bit different to what it would be as just a testhat expectation (multiple stack traces).

I'm not using R a lot these days, but I'm happy to shepherd this and help where I can.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants