Is every batch supposed to run every feature?
Closed this issue · 4 comments
I have a fairly large suite of tests that I want to be able to speed up by running in parallel batches, but I can't seem to figure out a way to get that to actually work using the parallel and/or serenity.batch... properties - is there something I'm doing wrong?
I tried to make a simplified example based on the latest stable version as specified by the serenity-core repo:
structure
both runners are the same
the features are not identical, but are highly similar
cucumber 'glue' steps
@Given("Something happens")
public void Something_happens ()
{
int batch = Integer.parseInt(System.getProperty("serenity.batch.number"));
System.out.println("Doing something for batch [" + batch + ']');
boolean something = Wait.on("Some sort of thing")
.until(step::doSomething);
assert something : "Something went wrong";
}
@Given("Something else happens")
public void Something_else_happens ()
{
int batch = Integer.parseInt(System.getProperty("serenity.batch.number"));
System.out.println("Doing something else for batch [" + batch + ']');
boolean somethingElse = Wait.on("Some OTHER sort of thing")
.until(step::doSomethingElse);
assert somethingElse : "Something else went wrong";
}
corresponding steps library
@Step
public boolean doSomething ()
{
int something = Integer.parseInt(System.getProperty("serenity.batch.number"));
return (something + 1) % 2 == 0;
}
@Step
public boolean doSomethingElse ()
{
int somethingElse = Integer.parseInt(System.getProperty("serenity.batch.number"));
return (somethingElse - 1) > 0;
}
contents of pom.xml
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<encoding>UTF-8</encoding>
<serenity.version>2.4.34</serenity.version>
<serenity.maven.version>2.4.34</serenity.maven.version>
<serenity.cucumber.version>2.4.34</serenity.cucumber.version>
<tags/>
<webdriver.base.url/>
<batches>2</batches>
</properties>
<dependencies>
<dependency>
<groupId>net.serenity-bdd</groupId>
<artifactId>serenity-core</artifactId>
<version>${serenity.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>net.serenity-bdd</groupId>
<artifactId>serenity-cucumber6</artifactId>
<version>${serenity.cucumber.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>net.serenity-bdd</groupId>
<artifactId>serenity-screenplay</artifactId>
<version>${serenity.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>net.serenity-bdd</groupId>
<artifactId>serenity-screenplay-webdriver</artifactId>
<version>${serenity.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>net.serenity-bdd</groupId>
<artifactId>serenity-ensure</artifactId>
<version>${serenity.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>1.0.13</version>
</dependency>
<dependency>
<groupId>org.assertj</groupId>
<artifactId>assertj-core</artifactId>
<version>3.6.2</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<!-- Kotlin Stuff -->
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<version>1.5.0</version>
<executions>
<execution>
<id>compile</id>
<phase>none</phase>
<goals>
<goal>compile</goal>
</goals>
<configuration>
<sourceDirs>
<sourceDir>${project.basedir}/src/main/kotlin</sourceDir>
<sourceDir>${project.basedir}/src/main/java</sourceDir>
</sourceDirs>
</configuration>
</execution>
<execution>
<id>test-compile</id>
<goals>
<goal>test-compile</goal>
</goals>
<configuration>
<sourceDirs>
<sourceDir>${project.basedir}/src/test/kotlin</sourceDir>
<sourceDir>${project.basedir}/src/test/java</sourceDir>
</sourceDirs>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>3.0.0-M5</version>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<version>3.0.0-M5</version>
<configuration>
<includes>
<include>**/*Runner.java</include>
</includes>
<systemPropertyVariables>
<webdriver.base.url>${webdriver.base.url}</webdriver.base.url>
<serenity.batch.size>0${batches}</serenity.batch.size>
<serenity.batch.number>0${surefire.forkNumber}</serenity.batch.number>
</systemPropertyVariables>
<parallel>classes</parallel>
<threadCount>${batches}</threadCount>
<forkCount>${batches}</forkCount>
<reuseForks>false</reuseForks>
</configuration>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<executions>
<execution>
<id>default-compile</id>
<phase>none</phase>
</execution>
<execution>
<id>default-testCompile</id>
<phase>none</phase>
</execution>
<execution>
<id>java-compile</id>
<phase>compile</phase>
<goals>
<goal>compile</goal>
</goals>
</execution>
<execution>
<id>java-test-compile</id>
<phase>test-compile</phase>
<goals>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>net.serenity-bdd.maven.plugins</groupId>
<artifactId>serenity-maven-plugin</artifactId>
<version>${serenity.maven.version}</version>
<configuration>
<tags>${tags}</tags>
</configuration>
<dependencies>
<dependency>
<groupId>net.serenity-bdd</groupId>
<artifactId>serenity-core</artifactId>
<version>${serenity.version}</version>
</dependency>
</dependencies>
<executions>
<execution>
<id>serenity-reports</id>
<phase>post-integration-test</phase>
<goals>
<goal>aggregate</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
'Wait' utility using Kotlin
class Wait
{
private lateinit var reason : String
private var created : Long = System.nanoTime()
private var patience : Long = 5
private var timedAs : TimeUnit = SECONDS
private fun expired () : Boolean = System.nanoTime() - created > NANOSECONDS.convert(patience, timedAs)
companion object
{
@JvmStatic
fun on (reason: String) : Wait
{
val wait = Wait()
wait.reason = reason
return wait
}
}
fun until (check: Supplier<Boolean>) : Boolean
{
var passed : Boolean
println("Waiting on: [$reason] for up to [$patience $timedAs]")
do passed = check.get()
while (!passed && !expired())
return passed
}
}
It runs both tests in both batches
Batches are for distributed execution, not parallel execution, and I don't know if they were ever used with Cucumber. Why don't you just use the Cucumber 6 parallel execution?
I'd be happy to take a look at it - I think I tried cucumbers built in methods in the past but that was on an older version.
Due to the nature of the software I'm testing (Java Swing-based instead of web), I'm not able to use parallel threads to run multiple tests per machine, but rather have to batch the tests out to a number of VM's that have been set up for that purpose.
I guess to be a little more specific - I have a Jenkins pipeline set up that programmatically generates the correct number of runner class files, based on the number of available VM's, before building and executing the test suite with the -Dbatches arg set to that value.
The driver I have to use to automate interactions with the client (UFT from MicroFocus) has to be initialized to the specific machine being used to execute the tests, which I was previously able to work around thanks to batches running in their own forked environment.
I'm more than happy to improve upon that process if there's a better way - it's just that there's unfortunately no easy way to differentiate between multiple copies of the client open on the same system. Attempting to run multiple tests in parallel (via UFT) on the same machine causes one or more of the involved tests to crash with script errors (as in an actual script error popup message from Windows).
I've been trying to make things works using threads, but I can't seem to find any way to make UFT cooperate... is there no way to make Serenity or Cucumber run the different batches in their own Runtimes, rather than as a bunch of threads sharing the same runtime?