Gradle TestSets plugin

A plugin for the Gradle build system that allows specifying test sets (like integration or acceptance tests). A test set is a logical grouping of a source set and related dependency configurations, tasks and artifacts.

❗	Version 2.x of this plugin is a complete rewrite in Kotlin, using some recent Gradle features for deferred configuration. As such, it requires Gradle 4.10 or later. If you are using an earlier version of Gradle, please refer to the 1.x versions of the plugin.

Quickstart

One of the most common use cases for this plugin is to separate integration tests from unit tests within the same project. Using a separate test set (instead of other mechanisms like JUnit tags) allows for a clean separation of the code, as well as a different set of library dependencies for both types of tests.

Add the following to your build.gradle file:

// The plugins block needs to be at the top of your build script
plugins {
    id 'org.unbroken-dome.test-sets' version '2.0.3'
}

testSets {
    integrationTest
}

Place your integration test code in src/integrationTest/java, and the unit tests (like before) in src/test/java.

To execute only the integration tests, run the integrationTest Gradle task:

./gradlew integrationTest

You can add dependencies that are only used in integration tests to the integrationTestImplementation configuration:

dependencies {
    // Wiremock will only be available in integration tests, but not in unit tests
    integrationTestImplementation 'com.github.tomakehurst:wiremock:2.19.0'
}

Usage

Applying the plugin

To use the TestSets plugin, include the following in your Gradle script:

build.gradle

plugins {
    id 'org.unbroken-dome.test-sets' version '2.0.3'
}

Prerequisites

The TestSets plugin is designed to work in conjunction with the java plugin, or other JVM language plugins that follow a similar structure. It has been tested to work with groovy, scala, and org.jetbrains.kotlin.jvm.

You will need to run Gradle 4.10 or higher with a JDK 8 or higher to use the plugin.

💡	If you want to understand in detail what the test-sets plugin does under the hood, it is recommended to revisit the explanation of the different dependency configurations used by the Java Plugin in the Gradle user manual.

Test Sets DSL

A test set is a logical grouping of the following:

a source set;
a set of associated dependency configurations;
a Test task to run the tests;
a Jar task to package the tests;
optionally, an artifact that can be published.

To create a new test set, declare it inside the testSets block in the project’s build.gradle file, like this:

testSets {
    integrationTest
}

In this example "integrationTest" is the name of the test set being created. As part of the process, the TestSets plugin will automatically create the following objects:

A source set named integrationTest;
A dependency configuration named integrationTestImplementation, which extends from "testImplementation";
A dependency configuration named integrationTestRuntimeOnly, which extends from "testRuntimeOnly";
A Test task named integrationTest which will run the tests in the set;
A Jar task named integrationTestJar which will package the tests.

Now you can place your integration test sources in src/integrationTest/java and run them with the integrationTest task.

💡

The dependency configurations integrationTestImplementation, integrationTestRuntimeOnly and so on are actually created by Gradle as companions to the integrationTest source set. The test sets plugin will automatically make each of them extend from the corresponding test*** configuration.

This means that you can define a dependency in testImplementation and have it available in your integration tests as well:

testSets { integrationTest }

dependencies {
    // These dependencies will be available in integration tests as well as unit tests
    testImplementation 'org.junit.jupiter:junit-jupiter-api:5.3.1'
    testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.3.1'

    // Use the integrationTest-specific configurations if you need a dependency only there
    integrationTestImplementation 'com.github.tomakehurst:wiremock:2.19.0'
}

💡	When using multiple test sets, you will have a separate `Test` task for each. The `tasks.withType` idiom is useful for applying common configuration to all of them: testSets { integrationTest } // Make all tests use JUnit 5 tasks.withType(Test) { useJUnitPlatform() }

Extending other test sets

A test set can extend from other test sets, inheriting all the corresponding dependency configurations.

testSets {
    fooTest
    barTest { extendsFrom fooTest }
}

This will make all the barTest* configurations extend from the corresponding fooTest* configurations, as if you had written:

configurations {
    barTestImplementation.extendsFrom fooTestImplementation
    barTestCompileOnly.extendsFrom fooTestCompileOnly
    barTestRuntimeOnly.extendsFrom fooTestRuntimeOnly
    barTestAnnotationProcessor.extendsFrom fooTestAnnotationProcessor
}

It does not mean, however, that the source (classes / resources) of the extended test set will be available to the extending test set. To accomplish this, you must additionally define a dependency on the source set’s output:

dependencies {
    fooTestImplementation sourceSets.barTest.output
}

You can also use test libraries (see below) to enable sharing code between your test sets.

Changing the directory name

For a source set named "myTest", the java plugin by default assumes the directories src/myTest/java and src/myTest/resources. A different directory name can be specified using the dirName on the test set, for example:

testSets {
    myTest { dirName = 'my-test' }
}

Which would change the source set’s java and resources directories to src/my-test/java and src/my-test/resources, respectively. This also works with any plugin (Groovy, Scala or Kotlin) that adds an extension to the SourceSet type.

Predefined Unit Test Set

The JVM plugins (java, groovy and so on) automatically define a source set named test to hold unit tests, testImplementation and testRuntimeOnly configurations to declare its dependencies, and a test task to run the tests.

This can be viewed as a test set that is already present, and in fact is available under the name unitTest. You can reference and even modify the unitTest test set, just like you would any other test set. For example, you could change the directory name for your unit tests to unit-test instead of test:

testSets {
    unitTest { dirName = 'unit-test' }
}

All new test sets implicitly extend the "unitTest" set.

Running Tests as Part of the Build

By default, the tests in a custom test set are not executed when you call gradle build. This is by design, because other types of tests are slower or more expensive to run than unit tests. In CI builds, running such tests is often modeled as a separate step in the build pipeline.

If you would like the tests of a test set to be run as part of every build, you can add a dependency from Gradle’s check task to the test set’s Test task:

testSets {
    integrationTest
}

check.dependsOn integrationTest

Test Libraries

Test libraries are special test sets that allow you to more cleanly factor out common support code that is used by multiple test sets. For example, if you have a test set named integrationTest, and created some custom assertion helpers that you would like to use from both unit and integration tests, you could place them in a test library:

import org.unbrokendome.gradle.plugins.testsets.dsl.TestLibrary

testSets {
    testCommon(TestLibrary)
    unitTest { imports testCommon }
    integrationTest { imports testCommon }
}

dependencies {
    // A test library's API dependencies will also be available in
    // importing test sets
    testCommonApi 'org.junit.jupiter:junit-jupiter-api:5.3.1'
    testCommonApi 'org.assertj:assertj-core:3.11.1'

    // A test library's implementation is "private", it will be available
    // at runtime but importing test sets cannot use it from their code
    testCommonImplementation 'com.google.guava:guava:27.0-jre'
}

In contrast to a standard test set, a test library makes a distinction between API and implementation dependencies, similar to the Java Library Plugin in Gradle (but within the same project).

Note that we use imports instead of extendsFrom to use a library, which has somewhat different semantics. integrationTest.imports(testCommon) adds the following connections:

integrationTestImplementation will extend from testCommonApi
integrationTestImplementation will have a dependency on the output of the testCommon source set
integrationTestRuntimeOnly will extend from testCommonRuntimeClasspath

Unlike extendsFrom, importing a test library will not inherit any compile-only or annotation processor dependencies.

💡	If you do not like the extra `import` statement for `TestLibrary`, you can also use the `createLibrary` method to create a library: testSets { createLibrary('testCommon') }

Publishing an artifact

Optionally, an artifact containing the classes and resources of a test set or test library can be added to the project’s output.

To activate this, simply set the createArtifact property of the test set to true:

testSets {
    integrationTest { createArtifact = true }
}

This will add the artifact <projectName>-integrationTest.jar to the project’s artifacts.

💡	Publishing artifacts is especially useful for test libraries, because it means that you can reuse your common test code not only in the same project, but also in other projects.

You can modify the classifier of the JAR file by setting the classifier property on the test set. By default, it is the name of the test set.

The following example publishes the unit tests as an artifact with the classifier tests:

testSets {
    unitTest {
        createArtifact = true
        classifier = 'tests'
    }
}

Kotlin DSL Support

As the plugin itself is written in Kotlin, it should work with the Gradle Kotlin DSL without problems.

To create a test set, use any of the common idioms from the Kotlin DSL:

import org.unbrokendome.gradle.plugins.testsets.dsl.TestLibrary

plugins {
    id("org.unbroken-dome.test-sets") version "2.0.3"
}

testSets {

    // use the creating construct
    val fooTest by creating { ... }

    // or the create() method
    create("barTest") { ... }

    // use TestLibrary::class as an argument to create a library
    val myTestLib by creating(TestLibrary::class)

    // unitTest is already defined, so we need to use getting
    val unitTest by getting {
        imports(myTestLib)
    }
}

The plugin also contains some extension functions to allow creating or configuring test sets by simply putting their name, similar to Groovy (you need to put the names in quotes, however):

import org.unbrokendome.gradle.plugins.testsets.dsl.TestLibrary

plugins {
    id("org.unbroken-dome.test-sets") version "2.0.3"
}

testSets {
    val myTestLib = "myTestLib"(TestLibrary::class)

    "fooTest"()

    "barTest" {
        imports(myTestLib)

        // You can also reference other test sets or test libraries by name
        extendsFrom("fooTest")
    }

    // unitTest is already present, but we can configure it in the same way
    "unitTest" { imports(myTestLib) }
 }

IDE Support

Neither Eclipse nor IntelliJ IDEA support the notion of multiple test sets per project / module, so what the plugin does is only a "best fit" so you can at least run the tests from your IDE.

Eclipse

When importing the Gradle project into Eclipse, the TestSets plugin will automatically add each test set’s dependencies to the classpath.

SourceSets that are generated for a test set are automatically mapped to source folders in Eclipse, without any further configuration. The plugin will try to mark each of these source folders as "test code" (the icon in the package explorer will have a slightly different shading).

However, source folders in eclipse are really just that, with no additional isolation, and no possibility to define an independent classpath. That means that tests from one test set will never be executed in isolation, which may become an issue if you have files of the same name (e.g. log4j2-test.xml) in different test sets.

Eclipse does not support different scopes for dependencies; all dependencies (main, test and additional test sets) are thrown into a shared "Gradle classpath container".

Eclipse also does not support Gradle’s concept of compile-only or runtime-only dependencies, so you will, for example, see even classes from runtime-only dependencies in code completion.

IntelliJ IDEA

If you’re using the test-sets plugin in IDEA, make sure to check the option "Create separate module per source set" when importing the Gradle project, or afterwards in your Gradle settings. This will allow IDEA to manage the dependencies independently for each source set.

fpavageau/gradle-testsets-plugin