Completed 0 tests without failure.
aleator opened this issue · 4 comments
Using smallcheck-1.0.2, the following doesn't manage to come up with any examples.
smallCheck 8 $ \f -> (f [True]::Bool)
Completed 0 tests without failure.
I think that for this case, "Completed .. without failure" is not a good way of reporting this. Shouldn't the test count as failure if no testcases can be generated?
The fact that no testcases were generated looks like a bug to me. I'll look into this a bit later.
(And sorry about the late response, somehow I missed this issue initially.)
This might have something to do with the generic instances. Here are few more series that fail.
> Prelude Test.SmallCheck.Series Data.Functor.Identity> list 4 (series ::Series Identity ([Bool] -> Bool))
[]
> Prelude Test.SmallCheck.Series Data.Functor.Identity> list 4 (series ::Series Identity (Int -> Bool))
[]
> Prelude Test.SmallCheck.Series Data.Functor.Identity> list 4 (series ::Series Identity ([Int] -> Bool))
Ok, I fixed the underlying bug (where no cases were generated) — thanks a lot for the report. I traced the bug to the very first versions of SmallCheck, but in the recent versions it just became easier to stumble upon.
Regarding your question
Shouldn't the test count as failure if no testcases can be generated?
— no, I don't think so. Test fails, by definition, if a counterexample is found. If there's no counterexample, there's no failure.
Let's say only one test case could be generated — is it a failure or not? Perhaps one test case for a huge search space doesn't tell you anything, so that's still a failure?
I don't think it's up to SmallCheck to make such decisions.
Great! Thanks!