A Question: Maven Failsafe <threadCount> instead of <forkCount> for parallel run
Naumansh opened this issue · 15 comments
Question
As in the shared example project from Cucuable Readme, we use as a parameter to specify the number of threads we want to run for parallel execution. The issue with forkCount approach is that it create separate jvm instances according to the number of thread count which impacts the execution resources and also static contents of the test code.
The general approach is to use tag which will run multi threads in same jvm but it doesn't work with cucable plugin and it runs a single test only instead of specified number of threads.
Disclaimer: I truly believe this question is not related to Maven.
A helpful response will be really appreciated in this context. Thanks.
As Cucable does not run the tests but only slices the test resources, this is in my opinion more a question of Maven.
If you want to run the tests in threads, you should probably not have one runner per scenario but one runner per thread containing a list of tests. Cucable also supports this if - you can set a desired number of runners via the appropriate property and all scenarios are distributed among them.
I hope I understand the problem correctly.
@laxersaz If parallel execution is completely in control of maven then why do we use below property in cucuable plugin:
scenarios
Also i agree maven runs and controls the execution but it does matter how are we configuring our key plugins in it. Now if you see that maven runs multi threads using instead of actually but it doesn't work while using our cucuable plugin. thats my basic question.
Maven runs the tests by using Surefire or Failsafe.
Cucable splits tests into smaller chunks and creates runners depending on your configuration (the scenarios mode splits every feature file into scenarios - so it is a splitting option, not a running option).
Have you tried my suggestion that I gave you in my last answer?
Here is what the config after your suggestion: But its not running anything in parallel
<plugin>
<groupId>com.trivago.rta</groupId>
<artifactId>cucable-plugin</artifactId>
<version>${cucable.version}</version>
<executions>
<execution>
<id>generate-test-resources</id>
<phase>generate-test-resources</phase>
<goals>
<goal>parallel</goal>
</goals>
</execution>
</executions>
<configuration>
<sourceRunnerTemplateFile>src/test/resources/configuration/CustomCreator.java</sourceRunnerTemplateFile>
<sourceFeatures>src/test/resources/features</sourceFeatures>
<generatedFeatureDirectory>${generated.feature.directory}</generatedFeatureDirectory>
<generatedRunnerDirectory>${generated.runner.directory}</generatedRunnerDirectory>
<parallelizationMode>scenarios</parallelizationMode>
<desiredNumberOfRunners>${threads}</desiredNumberOfRunners>
<includeScenarioTags>(${tags}) and not @wip</includeScenarioTags>
<logLevel>minimal</logLevel>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${maven.failsafe.plugin.version}</version>
<executions>
<execution>
<id>Run parallel tests</id>
<phase>integration-test</phase>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
<configuration>
<threadCount>${threads}</threadCount>
<testFailureIgnore>true</testFailureIgnore>
<disableXmlReport>true</disableXmlReport>
<argLine>-Dfile.encoding=UTF-8</argLine>
<excludes>
<exclude>**/IT*.java</exclude>
</excludes>
</configuration>
</plugin>
Does this run any tests?
yes it does. So i specified -Dthreads=2 and gave 3 tags to run. It executed 3 tags but in sequence and not in parallel.
So this setup should create two runner classes with multiple feature references inside.
I guess you have to try different combinations of options (also depending on the framework you are using), like perCoreThreadCount false.
I also noticed that you are not using the parallel option at all which might be the problem:
The most obvious one is by using the parallel parameter. The possible values depend on the test provider used. For JUnit 4.7 and onwards, this may be methods, classes, both, suites, suitesAndClasses, suitesAndMethods, classesAndMethods or all. As a prerequisite in JUnit tests, the JUnit runner should extend org.junit.runners.ParentRunner. If no runner is specified through the annotation @org.junit.runner.RunWith, the prerequisite is accomplished.
i tried parallel with methods and it doesnt work at all. Same is suggested on cucumber official documents.
Trying with classes mess up everything.
Since parallel doesnt support runner or features so not sure.
Can you point me to the official documents where this is mentioned?
Please see here
https://cucumber.io/docs/guides/parallel-execution/
If you see below comment on from cucumber github issues, maybe you can get it what configuration can work:
mpkorstanje commented 21 hours ago •
This is an opensource project so there are no promises, timetables or guarantees. It will be done when it is done.
It will work as expected with parallel execution within one instance of Cucumber. Because a static context is used for the before hooks different runners (e.g. from the temyers/cucumber-jvm-parallel-plugin) will interfere with each other when executed concurrently.
So effectively it will be the same as using JUnits 4 @Before/AfterClass annotated methods or JUnit 5s @Before/AfterAll annotated methods. Note that the cucumber-junit will play nicely with both JUnit 4 class rules and the @Before/AfterClass annotations.
Then I would try it without Cucable and let Cucumber handle the parallelization. Or use Cucable to generate one runner that includes all needed features.
But this is all I can answer at this point as this question goes way beyong the scope of Cucable.
yeah unfortunately cucumber parallelization doesnt work with our plugin approach. But its ok i have managed to manipulate the forked jvms.
Just another small question before i close the thread is, can we create a runner or multiple runners for a single scenario? like i want to run scenario @123 for 40 times. Is there any configuration that help achieve it ? Thanks.
Yes, just use numberOfTestRuns like so:
https://github.com/trivago/cucable-plugin#numberoftestruns
Can this be closed now, @Naumansh ?