junit-team/junit5

Beginning with 1.7.1 standalone console does not find tests on Java 8

Closed this issue · 8 comments

ndimitry commented

Steps to reproduce

We are launching tests with the following command:

java -cp ./junit-platform-console-standalone-1.7.1.jar:./jar-file-with-tests.jar org.junit.platform.console.ConsoleLauncher --select-package=root.package.for.all.tests --reports-dir /specified/path

jar-file-with-tests.jar does contain a lot of classes with methods marked with @Test annotation.

This command works fine when:

  • we use junit-platform-console-standalone version 1.7.0 both on Java 8 and Java 11
  • we use junit-platform-console-standalone version 1.7.1 or 1.8.0-M1 on Java 11

When launching 1.7.1 console on Java 8 the report states that 2 containers were found (2 containers found, 2 containers started and 2 containers successful), no tests are present in the report (all other values in the report are 0 (like 0 tests found))

Context

The testing project (jar-file-with-tests.jar) is built using Java 8.

Issue seems to occur both in Windows and Linux environments.

ndimitry commented

Colleagues use a framework based on JUnit 5 with some enhancements needed for our project. In case they use JUnit 5.7.1 (jupiter) they also faced the same problem: tests were not discovered on Java8.
So possibly this issue is connected with some code reused for both jupiter and platform.

marcphilipp commented

Potentially related to junit-team/junit5-samples#146

marcphilipp commented

Potentially caused by #2500

ndimitry commented

Metadata (MANIFEST.MF) of the testing project contains option
Multi-Release: true
Testing project contains no main class.
It was built with no module mentioned in the pom file (no module-info class is present). When I execute (on java11):

java -p jar-file-with-tests.jar --list-modules

I get no modules for the testing project (possibly I should change that).

sormuras commented

jar-file-with-tests.jar does contain a lot of classes with methods marked with @Test annotation.

Can you provide us that jar-file-with-tests.jar file in order to reproduce this issue you describe? Or better, create a minimal version of it?

Multi-Release: true and Java modules should not matter when running on Java 8. Btw, which exact version of Java 8 do you use?

sormuras commented

Setup

I could reproduce this bug using the following incredients/commands with JDK 17-ea+19:

package sample;
class SampleTests { @org.junit.jupiter.api.Test void sample() {} }
  • javac --release 8 --class-path junit-platform-console-standalone-1.7.0.jar -d classes SampleTests.java
  • jar --create --file tests.jar -C classes .
  • jar --list --file tests.jar
META-INF/
META-INF/MANIFEST.MF
sample/
sample/SampleTests.class

✔ Expected Output

.
+-- JUnit Jupiter [OK]                  
| '-- SampleTests [OK]                  
|   '-- sample() [OK]
'-- JUnit Vintage [OK]

❌ Wrong Output

+-- JUnit Jupiter [OK]
'-- JUnit Vintage [OK]

ConsoleLauncher 1.7.0

java -cp junit-platform-console-standalone-1.7.0.jar;tests.jar org.junit.platform.console.ConsoleLauncher --select-package sample

  • ✔ Java 8
  • ✔ Java 17

ConsoleLauncher 1.7.1

java -cp junit-platform-console-standalone-1.7.1.jar;tests.jar org.junit.platform.console.ConsoleLauncher --select-package sample

  • ❌ Java 8
  • ✔ Java 17

ConsoleLauncher 1.8.0-M1

java -cp junit-platform-console-standalone-1.8.0-M1.jar;tests.jar org.junit.platform.console.ConsoleLauncher --select-package sample

  • ❌ Java 8
  • ✔ Java 17
  • 👍1
ndimitry commented

@sormuras thank you for reproducing this issue!