Adding a notifier changes the behaviour of the test

[melezov]

The docs state that adding a notifier acts like "adding another listener", but on all recent versions of Specs2 it changes the way the tests are executed, making our CI "successfully pass" even though tests are crashing.

Example without notifier:

d:\Code\specs2-notifier-test>sbt "testOnly *"
[info] welcome to sbt 1.5.5 (AdoptOpenJDK Java 14.0.2)
[info] set current project to specs2-notifier-test (in build file:/D:/Code/specs2-notifier-test/)
[info] CrashTest
[error] x Crash test
[error]  'CRASH-TEST' != 'CRASH-DUMMY' (CrashTest.scala:7)
[info] Total for specification CrashTest
[info] Finished in 111 ms
[info] 1 example, 1 failure, 0 error
[error] Failed: Total 1, Failed 1, Errors 0, Passed 0
[error] Failed tests:
[error]         com.example.CrashTest
[error] (Test / testOnly) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 1 s, completed 10 Nov 2021, 11:19:41

Example with built-in ConsoleNotifier:

d:\Code\specs2-notifier-test>sbt "testOnly * -- notifier org.specs2.reporter.ConsoleNotifier"
[info] welcome to sbt 1.5.5 (AdoptOpenJDK Java 14.0.2)
[info] set current project to specs2-notifier-test (in build file:/D:/Code/specs2-notifier-test/)
[start]   CrashTest
[example] Crash test
[failure] Crash test 'CRASH-TEST' != 'CRASH-DUMMY'
[end]     CrashTest
[info] Passed: Total 0, Failed 0, Errors 0, Passed 0
[success] Total time: 1 s, completed 10 Nov 2021, 11:19:52

The reproducible itself is miniscule, but I'm attaching the project definition and the spec for completeness:
specs2-notifier-test.zip

(the original project uses a custom notifier, but the results are the same - successfully passing on errors)

[melezov]

For clarity on "present on recent versions", I've tested this on Scala 2.13.7 + Specs2 4.10-4.13 and on Scala 3.1.0 + Specs2 5.0.0-RC-18

[melezov]

I tried using Printer instead of Notifier, but it changes the behaviour as well.
Interestingly enough, printFinalStats is called immediately and not after the single test is ran.

[melezov]

OK, I think I figured it out.
It seems that we can get to the default behaviour by appending "console" to the args.
Adding "console" restores expected behaviour and makes SBT exit with 1 instead of 0.

I've found this in the docs: https://etorreborre.github.io/specs2/guide/SPECS2-5.0.0-RC-18/org.specs2.guide.JUnitXmlOutput.html but it only mentions console output, not the exit code of SBT due to the "success/error" of the test task.

[etorreborre]

Thanks for the report (and the reproducible example!). It indeed looks like some data is lost when using notifiers without console. I will check that.

[etorreborre]

I pushed the fix for 5.x. I will do the same on the 4.x branch a bit later.