Courgette-JVM

Courgette-JVM is an extension of Cucumber-JVM with added capabilities to run cucumber tests in parallel on a feature level or on a scenario level. It also provides an option to automatically re-run failed scenarios.

Key Features

All features can be executed in parallel on independent threads.
All scenarios can be executed in parallel on independent threads.
Automatic re-run of failed scenarios.
Requires only 1 annotated class to run all feature files in parallel.
Single report generation for all executed features including embedded files (Json and Html reports)
Single re-run file listing all failed scenarios that occured during parallel execution.
Supports Cucumber-JVM 4
Supports JUnit and TestNG
Can be used with Gradle and Maven.
Searchable and paginated Courgette-JVM Html Report which includes all step definitions, embedded screenshots, thrown exceptions, pie chart and Courgette run information.

Requirements

Java 8
Java 9 and 10 (supported from version 2.3.1)
Java 11 (supported from version 3.0.0)

Installation

Repository: jcenter

Maven

<repositories>
    <repository>
      <id>jcenter</id>
      <url>https://jcenter.bintray.com/</url>
    </repository>
</repositories>

<dependency>
  <groupId>io.github.prashant-ramcharan</groupId>
  <artifactId>courgette-jvm</artifactId>
  <version>3.2.0</version>
</dependency>

Gradle

repositories {
    jcenter()
}

compile 'io.github.prashant-ramcharan:courgette-jvm:3.2.0'

Included Dependencies

cucumber-core 4.2.0
cucumber-java 4.2.0
cucumber-java8 4.2.0
cucumber-junit 4.2.0
testng 6.14.3
jackson-databind 2.8.8

Usage

Example projects:

Courgette-JVM supports JUnit and TestNG to run cucumber features and scenarios in parallel. A JUnit runner class must be annotated with @RunWith(Courgette.class) and a TestNG runner class must extend TestNGCourgette.

threads : The number of concurrent threads to run cucumber features.
- Example: If you have 10 cucumber features and you use 6 threads, 6 features would first run in parallel then the following 4 features would run in parallel.
runLevel : Options are CourgetteRunLevel.FEATURE or CourgetteRunLevel.SCENARIO
- If set to feature level, all features would run in parallel. If set to scenario level, all scenarios would be run in parallel.
rerunFailedScenarios : If set to true, any failed scenario will be immediately re-run in the same thread. If the re-run succeeds, the initial failure will be ignored and not cause the build to fail.
rerunAttempts : The number of re-run attempts for a failed scenario. (rerunFailedScenarios must be set to true)
showTestOutput : If set to true, the output for each feature will be redirected to the current I/O source or destination.
reportTargetDir : Target directory where courgette-report is generated. Set to target by default.
cucumberOptions : The standard cucumber options for specifying feature paths, glue, tags etc..

Additional

At the end of the test run, a single report ( if included in the cucumberOptions ) listing all executed features and scenarios will be created in the specified report path. All embedded images will be placed in the images folder in the specified report path.
A courgette-rerun.txt file listing all failed scenarios will be created in the specified rerun plugin path or the target folder ( default )
A Courgette-JVM Html report will be created in the reportTargetDir (defaulted to the target directory).

JUnit Runner

@RunWith(Courgette.class)
@CourgetteOptions(
        threads = 10,
        runLevel = CourgetteRunLevel.SCENARIO,
        rerunFailedScenarios = true,
        rerunAttempts = 1,
        showTestOutput = true,
        reportTargetDir = "build",
        cucumberOptions = @CucumberOptions(
                features = "src/test/resources/features",
                glue = "steps",
                tags = {"@regression", "not @wip"},
                plugin = {
                        "pretty",
                        "json:build/cucumber-report/cucumber.json",
                        "html:build/cucumber-report/cucumber.html",
                        "junit:build/cucumber-report/cucumber.xml"},
                strict = true
        ))
public class RegressionTestSuite {
}

TestNG Runner

@Test
@CourgetteOptions(
        threads = 10,
        runLevel = CourgetteRunLevel.SCENARIO,
        rerunFailedScenarios = true,
        rerunAttempts = 1,
        showTestOutput = true,
        reportTargetDir = "build",
        cucumberOptions = @CucumberOptions(
                features = "src/test/resources/features",
                glue = "steps",
                tags = {"@regression", "not @wip"},
                plugin = {
                        "pretty",
                        "json:build/cucumber-report/cucumber.json",
                        "html:build/cucumber-report/cucumber.html"},
                strict = true
        ))
public class RegressionTestSuite extends TestNGCourgette {
}

Gradle Build Task

tasks.withType(Test) {
    systemProperties = System.getProperties()
    systemProperties.remove("java.endorsed.dirs") // needs to be removed from Java 9
}

// JUnit
task regressionSuite(type: Test) {
    include '**/RegressionTestSuite.class'

    outputs.upToDateWhen { false }
}

// TestNG
task regressionSuite(type: Test) {
    useTestNG()

    include '**/RegressionTestSuite.class'

    outputs.upToDateWhen { false }
}

Gradle Run Options

To override the hard-coded courgette options (threads, runLevel, rerunFailedScenarios, showTestOutput, reportTargetDir) set in the runner class, you can provide system properties to the gradle task.

gradle regressionSuite -Dcourgette.threads=2 -Dcourgette.runLevel=FEATURE -Dcourgette.rerunFailedScenarios=false -Dcourgette.showTestOutput=true -Dcourgette.reportTargetDir=build

To override the hard-coded cucumber options (tags, glue, plugin, name, junit) set in the runner class, you can provide comma separated system properties to the gradle task.

gradle regressionSuite -Dcucumber.tags="@regression, ~@bug" -Dcucumber.glue="steps, hooks"

To specify non standard VM options (-X options)

gradle regressionSuite -Dcourgette.vmoptions="-Xms256m -Xmx512m"

JUnit Callbacks

You can add global setup and tear-down code to your Courgette test runner using the @CourgetteBeforeAll and @CourgetteAfterAll annotations. For example:

@RunWith(Courgette.class)
@CourgetteOptions(/* Your Courgette options here... */)
public class RegressionTestSuite {
    @CourgetteBeforeAll
    public static void setUp() {
        System.out.println("I will run before any tests execute");
    }
    
    @CourgetteAfterAll
    public static void tearDown() {
        System.out.println("I will run after all of the tests execute");
    }
}

You can add any number of annotated methods to your test suite class. If you need your callbacks to run in a specific order, pass order to the annotation: @CourgetteBeforeAll(order = 2).

Limitations and Known Issues

JUnit test notifier is not updated when running features in the IDE during parallel test execution.
- Each feature is run using the Cucumber CLI and because of this JUnit is not notified off the test result. The workaround to this is the Courgette-JVM html report which lists all test passes, failures and re-runs. Alternatively, you can review the Cucumber reports or the results from the build tool.
When there's a failure in the feature and your runner is set to 'runLevel = CourgetteRunLevel.FEATURE' and 'rerunFailedScenarios = true', the re-run cucumber feature report will override the original cucumber feature report.
- Use CourgetteRunLevel.SCENARIO which resolves this issue and retains all results in the cucumber report.
The following error is returned: "java.io.IOException: Cannot run program "java": CreateProcess error=206, The filename or extension is too long" [Windows OS Only]
- This is a known Windows Create Process issue and is related to the classpath exceeding the maximum length of 32K characters. To workaround this, please upgrade to either Java 9/10/11 and Courgette 3.0.0 as Courgette uses the Java Platform Module System instead of the URL classloader.

Submitting Issues

For any issues or requests, please submit here

volkovs/courgette-jvm