All unit tests after a failing one are not executed
ufoscout opened this issue · 9 comments
After a test failure, all the remaining tests are always reported as successfully executed no matter the real test result.
Here a simple reproducer:
@Test
public void testOne() throws Throwable {
performTest();
}
@Test
public void testTwo() throws Throwable {
performTest();
}
void performTest() throws Throwable {
final Waiter waiter = new Waiter();
new Thread(new Runnable() {
@Override
public void run() {
// ------- HERE A FAILING ASSERTION -------
waiter.assertTrue(false);
waiter.resume();
}
}).start();
waiter.await(100);
}
Both the tests are supposed to fail, but instead only the first one fails. If additional tests are added to the same class, all of them will be marked as "passed" even if they fail. This can be easily reproduced adding
@Test
public void testThree() throws Throwable {
performTest();
}
to the previous example.
Due to this issue it is not really possible to know which tests are failing and which are really passing.
Hi @ufoscout - So far I'm not able to reproduce the issue - the tests run as expected (the tests committed above pass). How are you running the tests? Perhaps something else is going on...
Hi @jhalterman
I created a minimal project to reproduce the issue here:
https://github.com/ufoscout/concurrentunit-failing-tests
Clone the repository and execute "mvn clean test". You should get the following maven output:
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running com.mycompany.app.AppTest
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.05 sec <<< FAILURE!
testOne(com.mycompany.app.AppTest) Time elapsed: 0.008 sec <<< FAILURE!
java.lang.AssertionError: expected true
at net.jodah.concurrentunit.Waiter.fail(Waiter.java:151)
at net.jodah.concurrentunit.Waiter.assertTrue(Waiter.java:62)
at com.mycompany.app.AppTest$1.run(AppTest.java:26)
at java.lang.Thread.run(Thread.java:745)
Results :
Failed tests: testOne(com.mycompany.app.AppTest): expected true
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
As you can see there is only one failure but there should be two! In fact only testOne fails but even testTwo is wrong and it should fail.
Here a screenshot of the very same tests executed in Eclipse with exactly the same wrong behaviour, i.e. one test fails as expected but the other one incredibly passes:
https://raw.githubusercontent.com/ufoscout/concurrentunit-failing-tests/master/EclipseScreenshot-wrong-test-execution.png
Hi @ufoscout -
Running the code from your repo, both tests fail as expected:
Running com.mycompany.app.AppTest
Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.044 sec <<< FAILURE!
testOne(com.mycompany.app.AppTest) Time elapsed: 0.008 sec <<< FAILURE!
java.lang.AssertionError: expected true
at net.jodah.concurrentunit.Waiter.fail(Waiter.java:151)
at net.jodah.concurrentunit.Waiter.assertTrue(Waiter.java:62)
at com.mycompany.app.AppTest$1.run(AppTest.java:26)
at java.lang.Thread.run(Thread.java:722)
testTwo(com.mycompany.app.AppTest) Time elapsed: 0 sec <<< FAILURE!
java.lang.AssertionError: expected true
at net.jodah.concurrentunit.Waiter.fail(Waiter.java:151)
at net.jodah.concurrentunit.Waiter.assertTrue(Waiter.java:62)
at com.mycompany.app.AppTest$1.run(AppTest.java:26)
at java.lang.Thread.run(Thread.java:722)
Results :
Failed tests: testOne(com.mycompany.app.AppTest): expected true
testTwo(com.mycompany.app.AppTest): expected true
Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
I also tried running it on a separate machine and got the same result:
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running com.mycompany.app.AppTest
Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.13 sec <<< FAILURE!
Results :
Failed tests: testOne(com.mycompany.app.AppTest): expected true
testTwo(com.mycompany.app.AppTest): expected true
Tests run: 2, Failures: 2, Errors: 0, Skipped: 0
I'm not sure why it should fail for you and not for me. Which Java version are you using?
Hi @jhalterman
I have no idea why they both fail for you.
With a colleague of mine, we tried right now to run the same tests in three different computers and we always got the exactly same result (i.e. one test fails and one test passes).
This is the configuration of the three machines:
- Kubuntu linux 14.10 with openJDK 1.8.0_40 64bit
- Kubuntu linux 14.10 with openJDK 1.7.0_79 64bit
- Windows Seven with Oracle JDK 1.7.0_07 64bit
maven is 3.2.5 in all the machines.
Hi @jhalterman
It seems that, in the Windows Seven machine only, if we execute the tests multiple times (e.g. doing 20 times "mvn test"), sometime both the tests fail.
Could you try to execute them multiple times in your machine to see if the behaviour is coherent?
I just pushed a commit that may fix this issue. Please try the latest in master and let me know how it goes.
It works as expected now 👍
0.3.3 has been released. It should be in maven central shortly.
BTW - thanks for reporting this!