5.7.1 breaks Bazel sample
jdai8 opened this issue · 13 comments
On bb94997, the Bazel sample fails to discover any tests:
~/code/junit5-samples/junit5-jupiter-starter-bazel (remotes/origin/r5.7.1) % git rev-parse HEAD
bb94997206f599709b1f053ac2492375a5f8e02b
~/code/junit5-samples/junit5-jupiter-starter-bazel (remotes/origin/r5.7.1) % ./bazelisk.py test //... --test_output all
INFO: Analyzed 2 targets (0 packages loaded, 0 targets configured).
INFO: Found 1 target and 1 test target...
INFO: Elapsed time: 0.127s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
PASSED: //src/test/java/com/example/project:junit5-jupiter-starter-bazel-test (see /private/var/tmp/_bazel_jack_dai/7ea6705a9c37cce3b6db455ebdb8ee2c/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/src/test/java/com/example/project/junit5-jupiter-starter-bazel-test/test.log)
INFO: From Testing //src/test/java/com/example/project:junit5-jupiter-starter-bazel-test
==================== Test output for //src/test/java/com/example/project:junit5-jupiter-starter-bazel-test:
Thanks for using JUnit! Support its development at https://junit.org/sponsoring
╷
└─ JUnit Jupiter ✔
Test run finished after 29 ms
[ 1 containers found ]
[ 0 containers skipped ]
[ 1 containers started ]
[ 0 containers aborted ]
[ 1 containers successful ]
[ 0 containers failed ]
[ 0 tests found ]
[ 0 tests skipped ]
[ 0 tests started ]
[ 0 tests aborted ]
[ 0 tests successful ]
[ 0 tests failed ]
================================================================================
//src/test/java/com/example/project:junit5-jupiter-starter-bazel-test (cached) PASSED in 0.9s
Executed 0 out of 1 test: 1 test passes.
INFO: Build completed successfully, 1 total action
After checking out the previous commit 11914c4, it works:
~/code/junit5-samples/junit5-jupiter-starter-bazel (remotes/origin/r5.7.1~1) % g rev-parse HEAD
11914c4a2befbb4c823d0dff5843a1317cfcbb3c
~/code/junit5-samples/junit5-jupiter-starter-bazel (remotes/origin/r5.7.1~1) % ./bazelisk.py test //... --test_output all
INFO: Analyzed 2 targets (0 packages loaded, 0 targets configured).
INFO: Found 1 target and 1 test target...
INFO: Elapsed time: 0.111s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
PASSED: //src/test/java/com/example/project:junit5-jupiter-starter-bazel-test (see /private/var/tmp/_bazel_jack_dai/7ea6705a9c37cce3b6db455ebdb8ee2c/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/src/test/java/com/example/project/junit5-jupiter-starter-bazel-test/test.log)
INFO: From Testing //src/test/java/com/example/project:junit5-jupiter-starter-bazel-test
==================== Test output for //src/test/java/com/example/project:junit5-jupiter-starter-bazel-test:
Thanks for using JUnit! Support its development at https://junit.org/sponsoring
╷
└─ JUnit Jupiter ✔
└─ CalculatorTests ✔
├─ 1 + 1 = 2 ✔
└─ add(int, int, int) ✔
├─ 0 + 1 = 1 ✔
├─ 1 + 2 = 3 ✔
├─ 49 + 51 = 100 ✔
└─ 1 + 100 = 101 ✔
Test run finished after 100 ms
[ 3 containers found ]
[ 0 containers skipped ]
[ 3 containers started ]
[ 0 containers aborted ]
[ 3 containers successful ]
[ 0 containers failed ]
[ 5 tests found ]
[ 0 tests skipped ]
[ 5 tests started ]
[ 0 tests aborted ]
[ 5 tests successful ]
[ 0 tests failed ]
================================================================================
//src/test/java/com/example/project:junit5-jupiter-starter-bazel-test (cached) PASSED in 1.0s
Executed 0 out of 1 test: 1 test passes.
INFO: Build completed successfully, 1 total action
@jdai8 Thanks for reporting! Do you have time to debug to see what's going on?
@marcphilipp sure, although I could use some guidance. The sample essentially just invokes the console launcher with --select-package:
From the 5.7.1 changelog, junit-team/junit5#2500 seemed like the only possible culprit, but AFAIK this sample isn't using modules.
@jdai8 I could not reproduce this locally:
$ git checkout r5.7.1
Switched to branch 'r5.7.1'
Your branch is up to date with 'origin/r5.7.1'
$ ./bazelisk.py test //... --test_output all
INFO: Analyzed 2 targets (0 packages loaded, 0 targets configured).
INFO: Found 1 target and 1 test target...
INFO: Elapsed time: 0.163s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
PASSED: //src/test/java/com/example/project:junit5-jupiter-starter-bazel-test (see /private/var/tmp/_bazel_marc/543254cdda28f56bba4a66a0e7feffaa/execroot/__main__/bazel-out/darwin-fastbuild/testlogs/src/test/java/com/example/project/junit5-jupiter-starter-bazel-test/test.log)
INFO: From Testing //src/test/java/com/example/project:junit5-jupiter-starter-bazel-test
==================== Test output for //src/test/java/com/example/project:junit5-jupiter-starter-bazel-test:
Thanks for using JUnit! Support its development at https://junit.org/sponsoring
╷
└─ JUnit Jupiter ✔
└─ CalculatorTests ✔
├─ 1 + 1 = 2 ✔
└─ add(int, int, int) ✔
├─ 0 + 1 = 1 ✔
├─ 1 + 2 = 3 ✔
├─ 49 + 51 = 100 ✔
└─ 1 + 100 = 101 ✔
Test run finished after 136 ms
[ 3 containers found ]
[ 0 containers skipped ]
[ 3 containers started ]
[ 0 containers aborted ]
[ 3 containers successful ]
[ 0 containers failed ]
[ 5 tests found ]
[ 0 tests skipped ]
[ 5 tests started ]
[ 0 tests aborted ]
[ 5 tests successful ]
[ 0 tests failed ]
================================================================================
//src/test/java/com/example/project:junit5-jupiter-starter-bazel-test (cached) PASSED in 1.1s
Executed 0 out of 1 test: 1 test passes.
INFO: Build completed successfully, 1 total action
I've enabled test output on CI now as well and it works:
https://github.com/junit-team/junit5-samples/runs/2146954961?check_suite_focus=true#step:6:1267
Since I don't know what else to check, I'm closing this issue for now.
@jdai8 If you have any additional info as to why this might be failing for you, please let us know.
@marcphilipp After updating a project of mine to 5.7.1, I started experiencing the same issue as described here. I've downgraded to 5.7.0 for now. My test plan creation logic looks like:
val testLauncher = LauncherFactory.create()
val testPlan = testLauncher.discover(
LauncherDiscoveryRequestBuilder
.request()
.selectors(DiscoverySelectors.selectPackage(testPackageName))
.listeners(LauncherDiscoveryListeners.abortOnFailure(), LauncherDiscoveryListeners.logging())
.filters(filters: _*)
.configurationParameters(configParamsAdjusted.asJava)
.build())The filters are looking for a tag, but given that the diff between 5.7.0 and 5.7.1 are pretty small and PR #2531 seems to be the only discovery-related change, I am suspecting it is related to the package selector as opposed to the filters.
It seems the change was just to force a / at the end of the package name, and it seems that although I am using Scala -- which I initially suspected may be doing something unexpected to the names -- this should still be safe in my environment.
Let's assume testPackageName above is equal to my.package.prefix, I checked in my JAR file and it looks as expected:
± unzip -l my-jar-file.jar | grep "my.package.prefix" | grep "TestClass.class"
XXXXX 02-01-1980 00:00 my/package/prefix/TestClass.class
(package/jar/test class names have been changed but are representative)
Do you have any suggestions on how to go about further investigating this? I would be happy to provide more information to understand if this is something I am doing wrong on my end or a legitimate bug in 5.7.1.
@xkrogen Thanks for the info! Essentially we'd need a reproducer. Could you try changing junit5-jupiter-starter-bazel until it reliably reproduces the issue in a fork?
Hi @marcphilipp thanks for the quick response! I don't really know where to start investigating such an issue, especially as I know nothing about Bazel, so I don't think that will be a fruitful path for me. If you have suggestions on how I can perform further investigation within the context of my own project, such as logging to enable or areas of the code to step through in a debugger, I will be happy to help. Otherwise, I understand if you can't take any action at this time with the limited information available.
@jdai8 Any ideas?
Unfortunately, I haven't had time to look into this more. Looks like we're covered by junit-team/junit5#2600 though. I guess we had trouble reproducing this because I'm on Java 8 and it looks like CI is running Java 11.
Nice! Great to see we have a working, minimal reproduction sample over there.
A fix was recently applied. It's shipping in 5.8.0-SNAPSHOT (and soon in 5.7.2).
Would it make sense to extend the Bazel sample with a SNAPSHOT-consuming configuration?
Hmm, I'm not really a fan of SNAPSHOTs because they're not reproducible. For example, when trying to figure out which junit version broke this sample, it would have been impossible to bisect if we were depending on a snapshot.
5.7.2 is now released and resolves this issue.