Description

This project enables you to run performance tests on your Gradle project to compare the differences in performance between Kotlin K1 and K2 compilers.

The assemble task, used by default, builds your project using Kotlin versions 2.0.0-RC1 and 1.9.23 for performance tests.

Three test scenarios are covered to benchmark the performance:

clean build
incremental build with non-ABI changes
incremental build with ABI changes

After the build finishes, open the benchmarkResult.ipynb Kotlin Notebook to compare the results.

Project requirements

Ensure that the JAVA_HOME environment variable is set.
If your project involves Android development, make sure to set the ANDROID_HOME environmental variable as well.

You can change the Kotlin version of your project, by setting the kotlin_version Gradle property.

If your project uses version catalog, you can update the settings.gradle.kts file to override the Kotlin version using the following code:

versionCatalogs {
    create("libs") {
        val kotlinVersion = System.getenv("kotlin_version")
        if (kotlinVersion != null) {
            version("kotlin", kotlinVersion)
        }
    }
}

Configure project settings

You can modify your gradle.properties file to configure your project for performance tests:

Set project.path to configure the path to your project.
Set project.git.url and project.git.commit.sha to specify the URL and commit of a Git repository from which to clone your project.
Set project.git.directory to specify the directory where the project is stored.

Configure build scenarios

You can modify your gradle.properties file to configure specific scenarios for your project:

Set scenario.non.abi.changes to specify files for incremental builds with non-ABI changes. For multiple files, separate them with commas.
Set scenario.abi.changes to specify a single file path for incremental builds with ABI changes.
Set scenario.task to define the default build task; if not set, the assemble task will be used by default.

Create custom build scenarios

You can define custom build scenarios for your project by specifying them in your build.gradle.kts file. Set up your scenarios using the PerformanceTask.scenarios task input or configure a Gradle profiler scenario file using the PerformanceTask.scenarioFile task input.

For example, you can add the following configuration to your build.gradle.kts file to use custom scenarios:

import org.jetbrains.kotlin.k2.blogpost.PerformanceTask

// Defines custom scenarios
val customScenarios = listOf(Scenario(
    name = "new scenario",
    cleanTasks = listOf("clean_task_if_needed"),
    kotlinVersion = kotlinVersion,
    tasks = listOf("task_to_execute"),
    projectDir = project.projectDir,
    nonAbiChanges = listOf("path_to_file"),
    warmUpRounds = 5,
    executionRounds = 5,
))

// Registers performance tasks for Kotlin versions 2.0.0-RC1 and 1.9.23
val benchmark_2_0 = PerformanceTask.registerPerformanceTask(project, "benchmark_2_0", "2.0.0-RC1") {
    scenarios.set(customScenarios)
}

val benchmark_1_9 = PerformanceTask.registerPerformanceTask(project, "benchmark_1_9", "1.9.23") {
    scenarios.set(customScenarios)
}

// Registers a task to run all benchmarks
tasks.register("runBenchmarks") {
    dependsOn(benchmark_2_0, benchmark_1_9)
}

Run performance test

To run performance tests, run the following command in your terminal:

./gradlew runBenchmarks

Visualize results

You must have the Kotlin Notebook plugin installed in IntelliJ IDEA Ultimate to view the results.

To analyze the results:

Open the benchmarkResult.ipynb Kotlin Notebook file.
Run all code cells in the Kotlin Notebook using the Run All button to display and compare the produced results.

Troubleshooting

You might encounter new warnings in your builds using Kotlin 2.0.0. You can either resolve these warnings or disable the allWarningsAsErrors compiler option to continue without fixing them.

If dependencies verification is enabled, ensure that the dependencies for both Kotlin 1.9 and Kotlin 2.0 are correctly included in your project setup.

The JSON build report output type is available since Kotlin 2.0.0-RC1 and Kotlin 1.9.23. If you are using the kotlinDsl plugin in the buildSrc subproject, we recommend applying the kotlin("jvm") plugin as well. This prevents issues related to unrecognized output types and ensures compatibility across your project's configuration.

teodor-i/k2-performance-metrics