Sometimes, errors and failures during a test aren't caught.
markehammons opened this issue · 8 comments
If you look at my repository here you'll see that my tests are all passing for this commit. However, only 3 tests from BindingSpec were being reported on. After noticing this and investigating, I found that the ZLib test I had written was crashing and stopping the evaluation of the tests following it. This has been a frustratingly common situation with Munit, and I'm not sure if it's because of the nature of my code, or something that needs resolved in Munit.
Are you able to construct and provide a self-contained, minimal reproduction of the problem?
I will give it a shot.
@markehammons Did you manage to find some minimization?
@szymon-rd I haven't had time these past weeks to try. I will try to do that once I catch up.
Hi!
I got bitten by the same bug as in the linked issue. I reproduced it in an example project.
https://github.com/FloWi/munit-bug-experiment/blob/main/src/test/scala/example/HelloSpec.scala
In my original project I used Either.catchNonFatal wrong. Apparently it was a fatal exception that I tried to catch in an Either.
This is the output
before assertion in first test
before wrong use of Either.catchNonFatal
example.HelloSpec:
+ first test 0.005s
[info] Passed: Total 1, Failed 0, Errors 0, Passed 1
[success] Total time: 1 s, completed 02.07.2023, 16:34:35
As you can see, the 1st test is displayed correctly. The 2nd test is being executed (the println statement gets written to stdout) but not reported as a failure. It's not displayed at all. The 3rd test doesn't even get executed.
I would prefer mUnit to show the crashed test correctly and not silently hide it.
It seems that the bug has been introduced in the v1 branch of munit. I tried it with these dependencies
libraryDependencies ++= Seq(
"org.typelevel" %% "cats-core" % "2.9.0",
"org.scalameta" %% "munit" % "0.7.29" % Test,
)and it crashed as expected but reports the error.
before assertion in first test
before wrong use of Either.catchNonFatal
example.HelloSpec:
+ first test 0.003s
==> X example.HelloSpec.expected error running tests 0.001s java.lang.IllegalAccessError: asdf
at example.HelloSpec.$anonfun$new$4(HelloSpec.scala:14)
at cats.syntax.EitherObjectOps$.catchNonFatal$extension(either.scala:391)
at example.HelloSpec.$anonfun$new$3(HelloSpec.scala:14)
[error] Test suite example.HelloSpec failed with java.lang.IllegalAccessError: asdf.
[error] This may be due to the ClassLoaderLayeringStrategy (ScalaLibrary) used by your task.
When I use the latest milestone release, it just silently fails.
libraryDependencies ++= Seq(
"org.typelevel" %% "cats-core" % "2.9.0",
"org.scalameta" %% "munit" % "1.0.0-M8" % Test
)I hope it helps with the bug-hunt.