yschimke/roborazzi

Make JVM Android integration test visible 🤖📸

Roborazzi

Make JVM Android Integration Test Visible

Roborazzi now supports Robolectric Native Graphics (RNG) and enables screenshot testing.📣

To save the image, run recordRoborazziDebug task.

 ./gradlew recordRoborazziDebug

To view the changes in the image, run compareRoborazziDebug task. This way, the changes between the image and the one you are saving now will be saved as [original]_compare.png. And generate a json file that contains the diff information in build/test-results/roborazzi.

 ./gradlew compareRoborazziDebug

To verify the changes in the image, run verifyRoborazziDebug task. If the image is different from the one you are saving now, the test will fail.

 ./gradlew verifyRoborazziDebug

Why test with JVM instead of testing on Android?

Because when testing on a device, tests can fail easily due to the device environment, animations, etc. This affects the reliability of the test and ultimately, if the test fails, it cannot be fixed.

Why not Paparazzi?

Paparazzi is a great tool to see the actual display in the JVM.
Paparazzi relies on LayoutLib, Android Studio's layout drawing tool, which is incompatible with Robolectric. This is because they both mock the Android framework.
To run tests with Hilt and actually click on components, you need Robolectric.

Try it out

Available on Maven Central.

Add Robolectric

This library is dependent on Robolectric. Please see below to add Robolectric.

https://robolectric.org/getting-started/

To take screenshots, please use Robolectric 4.10 alpha 1 or later and please add @GraphicsMode(GraphicsMode.Mode.NATIVE) to your test class.

@GraphicsMode(GraphicsMode.Mode.NATIVE)

Apply Roborazzi Gradle Plugin

Roborazzi is available on maven central.

This plugin simply creates Gradle tasks record, verify, compare and passes the configuration to the test.

Define plugin in root build.gradle

plugins {
...
  id "io.github.takahirom.roborazzi" version "[write the latest vesrion]" apply false
}

Apply plugin in module build.gradle

plugins {
...
  id 'io.github.takahirom.roborazzi'
}

or

root build.gradle

buildscript {
  dependencies {
    ...
    classpath 'io.github.takahirom.roborazzi:roborazzi-gradle-plugin:[write the latest vesrion]'
  }
}

module build.gradle

apply plugin: 'io.github.takahirom.roborazzi'

Add dependencies

// Core functions
testImplementation("io.github.takahirom.roborazzi:roborazzi:[write the latest vesrion]")
// JUnit rules
testImplementation("io.github.takahirom.roborazzi:roborazzi-junit-rule:[write the latest vesrion]")

How to use

Take a screenshot manually

You can take a screenshot by calling captureRoboImage().

app/src/test/java/../ManualTest.kt

import androidx.compose.ui.test.junit4.createAndroidComposeRule
import androidx.test.core.app.ActivityScenario
import androidx.test.espresso.Espresso.onView
import androidx.test.ext.junit.runners.AndroidJUnit4
import org.junit.Test
import org.junit.runner.RunWith
import org.robolectric.annotation.GraphicsMode

// All you need to do is use the captureRoboImage function in the test!
import com.github.takahirom.roborazzi.captureRoboImage


// Tips: You can use Robolectric while using AndroidJUnit4
@RunWith(AndroidJUnit4::class)
// Enable Robolectric Native Graphics (RNG) 
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ManualTest { 
    @get:Rule
    val composeTestRule = createAndroidComposeRule<MainActivity>()

    @Test
    fun captureRoboImageSample() {
      // Tips: You can use Robolectric with Espresso API
      // launch
      ActivitySenario.launch(MainActivity::class.java)

      // Capture screen
      onView(ViewMatchers.isRoot())
          // If you don't specify a screenshot file name, Roborazzi will automatically use the method name as the file name for you.
          // The format of the file name will be as follows:
          // build/outputs/roborazzi/com_..._ManualTest_captureRoboImageSample.png
          .captureRoboImage()

      // Capture Jetpack Compose Node
      composeTestRule.onNodeWithTag("MyComposeButton")
          .onParent()
          .captureRoboImage("build/compose.png")

      // Capture small view on window
      onView(withId(R.id.button_first))
          .captureRoboImage("build/button.png")

      // move to next page
      onView(withId(R.id.button_first))
          .perform(click())

      val view: View = composeTestRule.activity.findViewById<View>(R.id.button_second)
      // Capture view on window
      view.captureRoboImage("build/manual_view_on_window.png")

      val textView = TextView(composeTestRule.activity).apply {
        text = "Hello View!"
        setTextColor(android.graphics.Color.RED)
      }
      // Capture view not on window
      textView.captureRoboImage("build/manual_view_without_window.png")

      // Capture Jetpack Compose lambda
      captureRoboImage("build/manual_compose.png") {
        Text("Hello Compose!")
      }

      val bitmap: Bitmap = createBitmap(100, 100, Bitmap.Config.ARGB_8888)
          .apply {
            applyCanvas {
              drawColor(android.graphics.Color.YELLOW)
            }
          }
      // Capture Bitmap
      bitmap.captureRoboImage("build/manual_bitmap.png") 
    }
}

Integrate to your GitHub Actions

It is easy to integrate Roborazzi to your GitHub Actions.

Add a job to store screenshots

name: store screenshots

on:
  push

env:
  GRADLE_OPTS: "-Dorg.gradle.jvmargs=-Xmx6g -Dorg.gradle.daemon=false -Dkotlin.incremental=false"

jobs:
  test:
    runs-on: macos-latest

    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-java@v3.9.0
        with:
          distribution: 'zulu'
          java-version: 19

      - name: Gradle cache
        uses: gradle/gradle-build-action@v2

      - name: test
        run: |
          # Create screenshots
          ./gradlew app:recordRoborazziDebug --stacktrace
      
      # Upload screenshots to GitHub Actions Artifacts
      - uses: actions/upload-artifact@v3
        with:
          name: screenshots
          path: app/build/outputs/roborazzi
          retention-days: 30

Add a job to verify screenshots

name: verify test

on:
  push

env:
  GRADLE_OPTS: "-Dorg.gradle.jvmargs=-Xmx6g -Dorg.gradle.daemon=false -Dkotlin.incremental=false"

jobs:
  test:
    runs-on: macos-latest

    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-java@v3.9.0
        with:
          distribution: 'zulu'
          java-version: 19

      - name: Gradle cache
        uses: gradle/gradle-build-action@v2

      # Download screenshots from main branch
      - uses: dawidd6/action-download-artifact@v2
        with:
          name: screenshots
          path: app/build/outputs/roborazzi
          workflow: test.yaml
          branch: main

      - name: verify test
        id: verify-test
        run: |
          # If there is a difference between the screenshots, the test will fail.
          ./gradlew app:verifyRoborazziDebug --stacktrace

      - uses: actions/upload-artifact@v3
        if: ${{ always() }}
        with:
          name: screenshot-diff
          path: app/build/outputs/roborazzi
          retention-days: 30

      - uses: actions/upload-artifact@v3
        if: ${{ always() }}
        with:
          name: screenshot-diff-reports
          path: app/build/reports
          retention-days: 30

      - uses: actions/upload-artifact@v3
        if: ${{ always() }}
        with:
          name: screenshot-diff-test-results
          path: app/build/test-results
          retention-days: 30

Generate gif automatically

@Test
fun captureRoboGifSample() {
  onView(ViewMatchers.isRoot())
    .captureRoboGif("build/test.gif") {
      // launch
      ActivityScenario.launch(MainActivity::class.java)
      // move to next page
      onView(withId(R.id.button_first))
        .perform(click())
      // back
      pressBack()
      // move to next page
      onView(withId(R.id.button_first))
        .perform(click())
    }
}

Automatically generate gif with test rule

With the JUnit test rule, you do not need to name the gif image, and if you prefer, you can output the gif image only if the test fails.

This test will output this file.

build/outputs/roborazzi/com.github.takahirom.roborazzi.sample.RuleTestWithOnlyFail_captureRoboGifSampleFail.gif

@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class RuleTestWithOnlyFail {
  @get:Rule
  val roborazziRule = RoborazziRule(
    captureRoot = onView(isRoot()),
    options = Options(
      onlyFail = true
    )
  )

  @Test
  fun captureRoboLastImageSampleFail() {
    // launch
    ActivityScenario.launch(MainActivity::class.java)
    // move to next page
    onView(withId(R.id.button_first))
      .perform(click())
    // should fail because the button does not exist
    // Due to failure, the gif image will be saved in the outputs folder.
    onView(withId(R.id.button_first))
      .perform(click())
  }
}

Compose Support

Test target

@Composable
fun SampleComposableFunction() {
  var count by remember { mutableStateOf(0) }
  Column(
    Modifier
      .size(300.dp)
  ) {
    Box(
      Modifier
        .testTag("MyComposeButton")
        .size(50.dp)
        .clickable {
          count++
        }
    )
    (0..count).forEach {
      Box(
        Modifier
          .size(30.dp)
      )
    }
  }
}

Test (Just add RoborazziRule)

@RunWith(AndroidJUnit4::class)
@GraphicsMode(GraphicsMode.Mode.NATIVE)
class ComposeTest {
  @get:Rule
  val composeTestRule = createAndroidComposeRule<ComponentActivity>()

  @get:Rule
  val roborazziRule = RoborazziRule(
    composeRule = composeTestRule,
    captureRoot = composeTestRule.onRoot(),
    options = RoborazziRule.Options(
      RoborazziRule.CaptureType.Gif
    )
  )

  @Test
  fun composable() {
    composeTestRule.setContent {
      SampleComposableFunction()
    }
    (0 until 3).forEach { _ ->
      composeTestRule
        .onNodeWithTag("MyComposeButton")
        .performClick()
    }
  }
}

RoborazziRule options

You can use some RoborazziRule options

class RoborazziRule private constructor(
  ...
) : TestWatcher() {
  /**
   * If you add this annotation to the test, the test will be ignored by roborazzi
   */
  annotation class Ignore

  data class Options(
    val captureType: CaptureType = CaptureType.Gif,
    /**
     * capture only when the test fail
     */
    val onlyFail: Boolean = false,
    /**
     * output directory path
     */
    val outputDirectoryPath: String = DEFAULT_ROBORAZZI_OUTPUT_DIR_PATH,
    val roborazziOptions: RoborazziOptions = RoborazziOptions(),
  )

  enum class CaptureType {
    /**
     * Generate last images for each test
     */
    LastImage,

    /**
     * Generate images for each layout change such as TestClass_method_0.png for each test.
     */
    AllImage,

    /**
     * Generate gif images for each test
     */
    Gif
  }

Roborazzi options

data class RoborazziOptions(
  val captureType: CaptureType = if (isNativeGraphicsEnabled()) CaptureType.Screenshot() else CaptureType.Dump(),
  val verifyOptions: VerifyOptions = VerifyOptions(),
  val recordOptions: RecordOptions = RecordOptions(),
) {
  sealed interface CaptureType {
    class Screenshot : CaptureType

    data class Dump(
      val takeScreenShot: Boolean = isNativeGraphicsEnabled(),
      val basicSize: Int = 600,
      val depthSlideSize: Int = 30,
      val query: ((RoboComponent) -> Boolean)? = null,
    ) : CaptureType
  }

  data class VerifyOptions(
    /**
     * This value determines the threshold of pixel change at which the diff image is output or not.
     * The value should be between 0 and 1
     */
    val resultValidator: (result: ImageComparator.ComparisonResult) -> Boolean
  ) {
    constructor(
      changeThreshold: Float = 0.01F,
    ) : this(ThresholdValidator(changeThreshold))
  }

  data class RecordOptions(
    val resizeScale: Double = 1.0
  )

Dump mode

If you are having trouble debugging your test, try Dump mode as follows.

LICENSE

Copyright 2023 takahirom
Copyright 2019 Square, Inc.
Copyright The Android Open Source Project

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.