Wrong JUnit reporting makes failed test to be treaded as passing
jotel opened this issue · 8 comments
Describe the bug
When test suite fail in lifecycle methods e.g. before/after these failures are ignored and not reported properly back to JUnit which makes a dangerous situation when failed tests are actually treated as passing
To Reproduce
Steps to reproduce the behavior:
- clone https://github.com/jotel/aborted-tetsts-example
- run
./gradlew check - Build is successful, but it should not be
Expected behavior
The build should fail and all suites should be present in build/test-results/test/
Screenshots
There are four suites and five tests in the sample build but only two of both of them are reported.
Additional context
Also ignored test in FunSuite marked ignore(...) are not properly reported as disabled.
At first, it seems to me, that runner reports an "aborted" test to JUnit, but that somehow does not make the suite fail.
I'm suspecting rootTestDescriptor in here:
https://github.com/helmethair-co/scalatest-junit-runner/blob/master/src/main/java/co/helmethair/scalatest/reporter/JUnitReporter.java#L56
this might give some clues:
https://github.com/scalatest/scalatest/blob/00b4f4a31602d08db5d8f1d9eafbdeccf3c72f02/scalatest/src/main/scala/org/scalatest/tools/JUnitXmlReporter.scala
I also think that is the reason. Thank you!
Hi @giurim, thanks for the fix... is the release with the change going to be published soon?
The central repository manager was prettyslow yesterday and it still gives weird errors. I try to figure out and push the release asap.
I will setup a JCenter account and try to release there