arquillian/smart-testing

Categorized strategy doesn't work with Surefire 2.21.0

bartoszmajsak opened this issue · 0 comments

When updating Smart Testing from 2.19.1 to 2.21.0, besides minor code changes in tests, categorized strategy is not working anymore (or at least there is one failing categorized test CategorizedTestSelectionFunctionalTests#should_run_test_with_categories_loader_and_service.

Results in:

java.lang.AssertionError: 
Actual and expected should have same size but actual size is:
 <4>
while expected is:
 <1>
Actual was:
<[TestResult{className='org.jboss.arquillian.core.impl.loadable.JavaSPIExtensionLoaderTestCase', testMethod='*', result=PASSED},
    TestResult{className='org.jboss.arquillian.core.impl.loadable.ServiceRegistryLoaderTestCase', testMethod='*', result=PASSED},
    TestResult{className='org.jboss.arquillian.core.impl.loadable.ServiceRegistryTestCase', testMethod='*', result=PASSED},
    TestResult{className='org.jboss.arquillian.core.impl.loadable.LoadableExtensionLoaderTestCase', testMethod='*', result=PASSED}]>
	at org.arquillian.smart.testing.ftest.categorized.CategorizedTestSelectionFunctionalTests.should_run_test_with_categories_loader_and_service(CategorizedTestSelectionFunctionalTests.java:59)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.arquillian.smart.testing.rules.TestBed$1.evaluate(TestBed.java:54)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

BTW it should be renamed from CategorizedTestSelectionFunctionalTests to CategorizedTestSelectionFunctionalTest.

Bumping Surefire to 2.21.0 in the dogfooding repository results with the same error.

Basic changes available in https://github.com/arquillian/smart-testing/tree/surefire_2.21.0