/Android-screenshot-testing-playground

A sample repo to introduce screenshot testing in Android with different libraries

Primary LanguageKotlinMIT LicenseMIT

part of blog posts

Android screenshot testing playground

snapshotVsUiTests

A sample repo to introduce screenshot testing in Android.
It contains a wide variety of examples written with different screenshot testing libraries for a better comparison among them. These examples include tests for screens like the one above (module :recyclerviewscreen)

Warning
It is configured with AGP 8.2.0-beta05, so it requires Android Studio Hedgehog or higher!

Sponsors

Thanks to Screenshotbot for their support!

By using Screenshotbot instead of the in-build record/verify modes provided by most screenshot libraries, you'll give your colleages a better developer experience, since they will not be required to manually record screenshots after every run, instead getting notifications on their Pull Requests.

Thanks to EmergeTools for their support!

Emerge automatically generates and diffs snapshots on your behalf, eliminating complicated CI setup with Emulators, file storage, and golden/diffing management.

Introduction

This repo showcases how to snapshot test Dialogs, ViewHolders, RecyclerViews, Activities, Fragments and Jetpack Compose!
And even better: It is ready for you to add your own examples and try screenshot testing with many libraries on your own!

In order to help find the desired examples, the app is modularized accordingly:

  1. :dialogs: Showcases how to screenshot test dialogs created with DialogBuilder from the android View system. Examples for Compose dialogs will be added as well
  2. :recyclerviewscreen: Contains screenshot tests for Activities, Fragments, ViewHolders and RecyclerViews.
  3. :lazycolumnscreen: Includes Jetpack Compose screenshot tests examples, as well as examples for Activities & Fragments.

Each of these modules contains submodules. Each submodule name corresponds to a screenshot testing library. You'll find screenshot test examples with that library in it.

As of August 2023, there are many screenshot testing libraries that facilitate automated screenshot testing, namely:

  1. Cashapp Paparazzi
  2. Dropbox Dropshots
  3. Shot from pedrovgs
  4. Roborazzi from takahirom
  5. Facebook screenshot-tests-for-android
  6. Ndtp Android-testify 1
  7. Snappy from QuickBird

All of them have their own pros and cons. The ultimate goal of this repo is to help you choose the libraries that better meet your project's needs!

In order to do that, it contains the same/similar examples but written with different libraries2:

  1. Paparazzi
  2. Dropshots
  3. Shot
  4. Roborazzi
  5. Android-testify

BONUS: It also contains examples of Cross-Library Screenshot Tests: the very same screenshot tests running with multiple libraries, namely: Paparazzi, Roborazzi, Shot, Dropshots & Android-Testify. For that it uses Android UI Testing Utils 2.0.0

You can read more about it in this blog post series:

  1. A World Beyond Libraries: Cross-Library Screenshot tests on android
  2. Write Once, Test Everywhere: Cross-Library Screenshot Testing with AndroidUiTestingUtils 2.0.0

    More screenshot test examples, as well as examples with other libraries will be continuously added.

1 Android-testify was started at Shopify and changed to Ndtp in summer 2022.

2 Paparazzi does not support screenshot tests...

  1. For Activities and Fragments, due to clashes with Robolectric, which is required by Activity/FragmentScenarios to run on the JVM.
  2. With Pseudolocales, due to layoutlib.

Table of Contents

Before you start...

The need for screenshot testing

I strongly recommend to have a look at my blog series on snapshot testing

If reading is not your thing, you can always watch my 2021 Droidcon tech-talks on the matter:

  1. Droidcon Berlin 2021
  2. Droidcon London 2021

Roborazzi tests on Windows

If you are running screenshot tests on a Windows machine, you must know that Roborazzi tests will fail. That's because it is built based on Robolectric Native Graphics (RNG), which does not support Windows currently. This issues has already been reported here.

Emulators

For instrumented screenshot testing (which excludes Paparazzi), I've been using emulators running API 27-31. Moreover, if you are running screenshot tests on a Windows machine, beware that Shot had some issues in the past, although they should be already solved.

Animations

Instrumented screenshot tests might have issues due to running animations at the moment the snapshot are taken.

In order to keep animations disabled, this project uses testOptions { animationsDisabled = true } in the gradle file. Unfortunately, it does not work on all APIs (e.g. API 26 or lower). Therefore, if you come across some issues, disable animations on the emulator via settings before running the screenshot tests.

Comparing screenshot testing libraries

Paparazzi vs. on-device screenshot testing libraries

Need for emulators

Roborazzi & Paparazzi let you run screenshot tests on the JVM, without emulators/devices. I'm still evaluating Roborazzi, so this section refers to Paparazzi only, and will be extended in the future.

Although running screenshot tests on the JVM comes with some speed wins, its main advantage is that one doesn't have to deal with emulator and their problems, such as:

  1. Emulators eventually freezing/crashing (specially on CI)
  2. "Insufficient storage" errors

Rendering elevation in generated screenshots

Paparazzi uses PixelCopy to generate bitmaps out of Views. Most on-device screenshot testing frameworks use Canvas as default1 to generate bitmaps, what ignores elevation.

This is specially noticeable in API 31:

However, on-device screenshot testing libraries also accept bitmaps as arguments of their take/verify screenshot methods.

// Shot
compareScreenshot(
    bitmap = pixelCopyBitmapFromView
)
// Dropshots
dropshots.assertSnapshot(
    bitmap = pixelCopyBitmapFromView
)

Therefore, they could render elevation by converting views to bitmaps using PixelCopy. Android UI Testing Utils provides the drawToBitmapWithElevation() method for that.

And the resulting screenshot would render elevation

You can find such examples in this repo, under the bitmap folder under any :dropshots, :shot and :android-testify module.

1 Shot doesn't use Canvas when using compareScreenshot(composeRule), so those screenshots draw elevation. It does use Canvas for Views, Activities, Fragments & Dialogs though.

Screenshot testing Activites and Fragments

ActivityScenarios and FragmentScenarios are compatible with Robolectric, which stubs the Android framework to run instrumented tests on the JVM. Therefore, we could also use Robolectric to run Activity/Fragment screenshot tests on the JVM with Paparazzi, theoretically. However, it crashes in doing so with some Byte Buddy exception at runtime, likely due to some clashes with Robolectric.

Roborazzi, on the other hand, is built on top of Robolectric, so ActivityScenarios and FragmentScenarios are compatible with it.

This means, only on-device screenshot testing frameworks & Roborazzi can be used for snapshoting Activites/Fragments. Only Paparazzi cannot.

Rendering problems

Paparazzi relies on layoutlib to record screenshots. That's a private library used to render the xml layouts and Compose previews in Android Studio. This comes with some limitations. For example,

  1. Composables using NavHost cannot be rendered in the @Preview, and therefore, Paparazzi cannot either, for now.
  2. Renders incorrectly UI elements that use multiple View.animate() and/or ObjectAnimator.ofPropertyValuesHolder() for animations. You can check it out yourself by running ./gradlew :recyclerviewscreen:paparazzi:recordPaparazziDebug in this repo. For instance:
View.animate() View.animate() + ObjectAnimator
  1. Doesn't support Pseudolocales. We can use pseudolocales to detect layout alignment issues without the need to render the screen in all languages our app supports. If set while testing, such Paparazzi tests crash. You can read more about pseudolocales in the official android documentation.

On the other hand, on-device screenshot testing has its own issues as well. Most of them happen due to the hardware accelerated drawing model. This happens for example, when running the same test on machines with different architectures1 e.g. Macbook Pro with M1 chip vs. Intel x64. It might cause issues related to:

  1. Shadows & elevation
  2. Font smoothing & anti-aliasing
  3. Image decompression and rendering
  4. Alpha-blending.

Nevertheless, you can solve/mitigate such issues as follows:

  1. Hardware acceleration can be enabled/disabled at different levels e.g. Activity, Window, View... to reduce such issues. Keep in mind that this affects how the screenshots are rendered.
  2. Most libraries provide a "tolerance/maxPixelDiff" mechanism to set some error threshold when verifying the screenshot. Although it isn't a real fix, you could use it to mitigate such issues. There is an interesting blog on android-testify about this.
  3. The safest solution is to generate the screenshots only on the CI, then pushing them in a new commit on your PRs. Same goes for "verify". This ensures that the screenshots are always generated using the same hardware.

Read more about hardware acceleration in the official Android documentation

1 There is evidence of such problems in Paparazzi as well, when running tests on different operating systems, as stated here

Stability: Android Gradle Plugin and Compose runtime updates

Unfortunately, any Android Gradle Plugin (a.k.a. AGP) or Compose update can make your working Paparazzi screenshot tests break... or fix those broken. For instance:

  1. When updating to Compose runtime 1.4.x: this issue & the corresponding fix
  2. Before updating AGP to that required by Android Studio Dolphin: Compose Dialog rendering issue

Summary: Pros and Cons

Let's summarize.

Pros

  1. No emulators needed.
    1. Faster
    2. No emulator troubleshooting
  2. Uses PixelCopy by default to generate bitmaps out of the views. Thus, screenshots render UI elements with elevation (e.g. shadows)

Cons

  1. Cannot screenshot Activities or Fragments
  2. Rendering problems
    1. Incorrect screenshots for UI components that call View.animate() or ObjectAnimator.ofPropertyValuesHolder() several times.
    2. Only renders what the Compose @Previews can display
  3. Fragile to AGP & Jetpack Compose updates
  4. No support for Pseudolocales

Recording and verifying screenshots

For screenshot testing, 2 tasks are required:

  1. Record. When executed, it generates a snapshot file for each test that will be taken as reference.
  2. Verify. When executed, it generates a snapshot file for each test that will be compared, pixel by pixel, with the one taken as reference when recording.

All Screenshot testing frameworks provide at least these 2 tasks.

Warning
All the commands in the description are for MacOS. You might need to adjust them depending on the operating system of your machine.

On-device tests with Android Orchestrator

By default, Android Orchestrator is disabled. That's because it makes the screenshot test run much slower, while adding no value. Android orchestrator might make sense when running screenshot test that might influence each other. For instance, when screenshot testing Activities or Fragments where we fetch some data.

For screenshot test that inflate a view and populate it with some data before the snapshot, it makes little sense.

However, you can run Dropshots, Shot and Android-testify screenshot tests with Android Orchestrator in this repo by adding the following to the corresponding command: -PuseOrchestrator

For instance, to record with Dropshots: ./gradlew :dialogs:dropshots:connectedAndroidTest -Pdropshots.record -PuseOrchestrator

Paparazzi

No emulators required. Run the following gradle tasks depending on the module:

  1. Record: ./gradlew :module_name:paparazzi:recordPaparazziDebug. For instance:
    1. ./gradlew :dialogs::paparazzi:recordPaparazziDebug
    2. ./gradlew :recyclerviewscreen:paparazzi:recordPaparazziDebug
    3. ./gradlew :lazycolumnscreen:paparazzi:recordPaparazziDebug
  2. Verify: ./gradlew :module_name:paparazzi:verifyPaparazziDebug. For instance:
    1. ./gradlew :dialogs:paparazzi:verifyPaparazziDebug
    2. ./gradlew :recyclerviewscreen:paparazzi:verifyPaparazziDebug
    3. ./gradlew :lazycolumnscreen:paparazzi:verifyPaparazziDebug

Note
You can record/verify the tests in parallel with the gradle property -Pparallel e.g. ./gradlew :module_name:paparazzi:recordPaparazziDebug -Pparallel Please note that running tests in parallel is only worthwhile when dealing with a large number of tests.

Roborazzi

No emulators required. Run the following gradle tasks depending on the module:

  1. Record: ./gradlew :module_name:roborazzi:recordRoborazziDebug. For instance:
    1. ./gradlew :dialogs::roborazzi:recordRoborazziDebug
    2. ./gradlew :recyclerviewscreen:roborazzi:recordRoborazziDebug
    3. ./gradlew :lazycolumnscreen:roborazzi:recordRoborazziDebug
  2. Verify: ./gradlew :module_name:roborazzi:verifyRoborazziDebug. For instance:
    1. ./gradlew :dialogs:roborazzi:verifyRoborazziDebug
    2. ./gradlew :recyclerviewscreen:roborazzi:verifyRoborazziDebug
    3. ./gradlew :lazycolumnscreen:roborazzi:verifyRoborazziDebug

In order to see the screenshots in Android Studio, change the view from "Android" to "Project".

Note 1
Thanks to Roborazzi plugin, it is also possible to record/verify these tests directly from Android Studio. For recording, you need to add roborazzi.test.record=true in your gradle.properties file. For verifying, you need to add roborazzi.test.verify=true in your gradle.properties file. Remember to record before verifying, to have a reference to compare with.

Note 2
You can record/verify the tests in parallel with the gradle property -Pparallel e.g. ./gradlew :module_name:roborazzi:recordRoborazziDebug -Pparallel Please note that running tests in parallel is only worthwhile when dealing with a large number of tests.

Dropshots

Start the emulator. Then run the following gradle tasks depending on the module:

  1. Record: ./gradlew :module_name:dropshots:connectedAndroidTest -Pdropshots.record. For instance:
    1. ./gradlew :dialogs:dropshots:connectedAndroidTest -Pdropshots.record
    2. ./gradlew :recyclerviewscreen:dropshots:connectedAndroidTest -Pdropshots.record
    3. ./gradlew :lazycolumnscreen:dropshots:connectedAndroidTest -Pdropshots.record
  2. Verify: ./gradlew :module_name:dropshots:connectedAndroidTest. For instance:
    1. ./gradlew :dialogs:dropshots:connectedAndroidTest
    2. ./gradlew :recyclerviewscreen:dropshots:connectedAndroidTest
    3. ./gradlew :lazycolumnscreen:dropshots:connectedAndroidTest

Note
Thanks to Dropshots plugin, it is also possible to record/verify these tests directly from Android Studio. By default, it will execute the verify task. For recording, you need to add dropshots.record in your gradle.properties file. Remember to record before verifying, to have a reference to compare with.

Shot

Start the emulator. Then run the following gradle tasks depending on the module:

  1. Record: ./gradlew :module_name:shot:executeScreenshotTests -Precord. For instance:
    1. ./gradlew :dialogs:shot:executeScreenshotTests -Precord
    2. ./gradlew :recyclerviewscreen:shot:executeScreenshotTests -Precord
    3. ./gradlew :lazycolumnscreen:shot:executeScreenshotTests -Precord
  2. Verify: ./gradlew :module_name:shot:executeScreenshotTests. For instance:
    1. ./gradlew :dialogs:shot:executeScreenshotTests
    2. ./gradlew :recyclerviewscreen:shot:executeScreenshotTests
    3. ./gradlew :lazycolumnscreen:shot:executeScreenshotTests

Note
The library says the record reports can be reviewed at RoadToEffectiveSnapshotTesting/dialogs/shot/build/reports/shot/debug/index.html However, it is wrong. The record reports can be reviewed at RoadToEffectiveSnapshotTesting/dialogs/shot/build/reports/shot/debug/record/index.html The path for the verification reports is right though.

Android-Testify

Currently this is the only library that supports Gradle Managed Devices. You can do it by directly running the following command:

  1. Record: ./gradlew :module_name:android-testify:pixel3api30DebugAndroidTest -PrecordModeGmd. For instance:
    1. ./gradlew :dialogs:android-testify:pixel3api30DebugAndroidTest -PrecordModeGmd
    2. ./gradlew :recyclerviewscreen:android-testify:pixel3api30DebugAndroidTest -PrecordModeGmd
    3. ./gradlew :lazycolumnscreen:android-testify:pixel3api30DebugAndroidTest -PrecordModeGmd
  2. Verify: ./gradlew :module_name:android-testify:pixel3api30DebugAndroidTest. For instance:
    1. ./gradlew :dialogs:android-testify:pixel3api30DebugAndroidTest
    2. ./gradlew :recyclerviewscreen:android-testify:pixel3api30DebugAndroidTest
    3. ./gradlew :lazycolumnscreen:android-testify:pixel3api30DebugAndroidTest

Note
Before verifying, you need to copy the files saved under the corresponding module's build/outputs/managed_device_android_test_additional_output/...) to the correct location, as specified here: https://ndtp.github.io/android-testify/docs/recipes/gmd

If you do not want to use Gradle Managed Devices, do as below: First, start the emulator. Then run the following gradle tasks depending on the module:

  1. Record: ./gradlew :module_name:android-testify:screenshotRecord. For instance:
    1. ./gradlew :dialogs:android-testify:screenshotRecord
    2. ./gradlew :recyclerviewscreen:android-testify:screenshotRecord
    3. ./gradlew :lazycolumnscreen:android-testify:screenshotRecord
  2. Verify: ./gradlew :module_name:android-testify:screenshotTest. For instance:
    1. ./gradlew :dialogs:android-testify:screenshotTest
    2. ./gradlew :recyclerviewscreen:android-testify:screenshotTest
    3. ./gradlew :lazycolumnscreen:android-testify:screenshotTest

Cross-Library

Run the very same screenshot tests with the screenshot testing library of your choice, among Paparazzi, Roborazzi, Shot, Dropshots or Android-Testify.
The most common scenario is to use 1 on-device (e.g. Shot, Dropshots, Android-Testify) and 1 JVM (e.g. Paparazzi, Roborazzi) screenshot library. For that, check the corresponding submodules e.g. :dropshots+paparazzi, :shot+roborazzi, :dropshots+roborazzi
You would execute such tests as you would do for the corresponding library e.g. ./gradlew :lazycolumnscreen:dropshots+paparazzi:recordPaparazziDebug would record with Paparazzi.

For examples containing 2+ on-device and/or 2+ JVM screenshot libraries, check the corresponding :crosslibrary modules.
Since they configure 2 on-device & 2 JVM screenshot libraries, you need to pass the library name via command line for its correct execution:

  1. Record:
    1. Paparazzi: ./gradlew :lazycolumnscreen:crosslibrary:recordPaparazziDebug -PscreenshotLibrary=paparazzi
    2. Roborazzi: ./gradlew :lazycolumnscreen:crosslibrary:recordRoborazziDebug -PscreenshotLibrary=roborazzi
    3. Shot: ./gradlew :lazycolumnscreen:crosslibrary:executeScreenshotTests -Precord -PscreenshotLibrary=shot
    4. Dropshots: ./gradlew :lazycolumnscreen:crosslibrary:connectedAndroidTest -Pdropshots.record -PscreenshotLibrary=dropshots
    5. Testify: ./gradlew :lazycolumnscreen:crosslibrary:screenshotRecord -PscreenshotLibrary=android-testify
  2. Verify:
    1. Paparazzi: ./gradlew :lazycolumnscreen:crosslibrary:verifyPaparazziDebug -PscreenshotLibrary=paparazzi
    2. Roborazzi: ./gradlew :lazycolumnscreen:crosslibrary:verifyRoborazziDebug -PscreenshotLibrary=roborazzi
    3. Shot: ./gradlew :lazycolumnscreen:crosslibrary:executeScreenshotTests -PscreenshotLibrary=shot
    4. Dropshots: ./gradlew :lazycolumnscreen:crosslibrary:connectedAndroidTest -PscreenshotLibrary=dropshots
    5. Testify: ./gradlew :lazycolumnscreen:crosslibrary:screenshotTest -PscreenshotLibrary=android-testify

Note
You can also record and verify via Gradle Managed Devices with Android-Testify as specified in the previous section:

  1. Record: ./gradlew :lazycolumnscreen:crosslibrary:pixel3api30DebugAndroidTest -PrecordModeGmd
  2. Verify: ./gradlew :lazycolumnscreen:crosslibrary:pixel3api30DebugAndroidTest specified here: https://ndtp.github.io/android-testify/docs/recipes/gmd

To enable cross-library screenshot testing, it uses Android UI Testing Utils 2.1.0

Parameterized Screenshot Tests

Some tests in this repo are written as parameterized. In doing so, we can test a given view with all possible different configurations, namely:

  1. UI mode (Light and Dark)
  2. Font size
  3. Locale
  4. Orientation
  5. Custom themes
  6. Display size
  7. View dimensions (width and/or height)
  8. Specific view states
  9. Others...

With Parameterized snapshot test we can write the test once and run it for all the desired configurations! You can read more about how we can profit from it here:

This can be achieved by using either org.junit.runners.Parameterized, com.google.testing.junit.testparameterinjector.TestParameterInjector or in case of Roborazzi via org.robolectric.ParameterizedRobolectricTestRunner, as you'll find in most examples in this repo.

Dropshots & Shot do not offer the possibility to set such configurations; Android-Testify only partially; Paparazzi and Roborazzi (via Robolectric) do though. Nevertheless, we can set the desired configurations for those libraries with Android UI Testing Utils, as seen in the examples.

Remark The Parameterized runner injects the parameter passed in the Test class constructor to every single test. If that is not wished, consider using the TestParameterInjector. On the other hand, TestParameterInjector only works with instrumented test on API 24+. Keep that in mind.

Filtered parameterized screenshot tests

Running snapshot tests for all configurations on every PR can be very time-consuming and lead to incrementally slower builds. One approach to solve that issue is to run only a part of those tests on every PR (e.g. the most common config, our "smoke/happy path" tests) and all snapshot tests - or "all non-smoke tests" - once a day (e.g. during the night or whenever it disturbs your team the least). In doing so, we get informed of the most important visual regression bugs on every PR (i.e. blocking bugs), and still get notified of the non-blocking bugs once a day.

We can accomplish this by filtering our tests accordingly.

Instrumented tests

This applies for tests running on emulators/physical devices i.e. those using Dropshots, Shot or Android-Testify

In order to filter tests, we need to provide the corresponding test instrumentation arguments. A straightforward means to do that is to use custom annotations. For instance:

  1. -Pandroid.testInstrumentationRunnerArguments.annotation=com.your.package.YourAnnotation
  2. -Pandroid.testInstrumentationRunnerArguments.notAnnotation=com.your.package.YourAnnotation

Warning
These arguments are supported by org.junit.runners.Parameterized and com.google.testing.junit.testparameterinjector.TestParameterInjector, but not by all runners, e.g. JUnitParams fails if used. I strongly recommend to use com.google.testing.junit.testparameterinjector.TestParameterInjector for filtered parameterized tests, because you can control which parameters are passed wo each test method. That is not the case with org.junit.runners.Parameterized, because the parameters are injected into every single test method. Thus, you need to create different Test classes to inject different parameters.

That's why you'll find some tests in this repo with the following annotations, which are found in the module testannotations:

  1. @UnhappyPath
  2. @HappyPath

In order to run filtered tests, execute the following gradle tasks, for the given annotation, for example,@UnhappyPath with shot:

  1. Record: ./gradlew :module_name:shot:executeScreenshotTests -Pandroid.testInstrumentationRunnerArguments.annotation=com.example.road.to.effective.snapshot.testing.testannotations.UnhappyPath -Precord
  2. Verify: ./gradlew :module_name:shot:executeScreenshotTests -Pandroid.testInstrumentationRunnerArguments.annotation=com.example.road.to.effective.snapshot.testing.testannotations.UnhappyPath

The same goes for @HappyPath

Gradle tests

The previous approach only works for instrumented tests. For non-instrumented tests, like those running with Paparazzi or Roborazzi, the simplest approach is to use gradle test filters. We can pass them as parameters when executing the corresponding paparazzi test tasks. For this to work, it is important to define a clear test naming convention, for example:

  1. UnhappyPath tests -> inside class myTestClassNameUnhappyPathTest
  2. HappyPath tests -> inside class myTestClassNameHappyPathTest

This enables to filter the test separately by executing:

  1. Only Happy path tests:
    1. Paparazzi: ./gradlew :module_name:paparazzi:recordPaparazziDebug --tests '*UnhappyPath*'
    2. Roborazzi: ./gradlew :module_name:roborazzi:recordRoborazziDebug --tests '*UnhappyPath*'
  2. Only Unhappy path tests:
    1. Paparazzi: ./gradlew :module_name:paparazzi:recordPaparazziDebug --tests '*HappyPath*'
    2. Roborazzi: ./gradlew :module_name:roborazzi:recordRoborazziDebug --tests '*HappyPath*'

Note
This approach does not work for instrumented tests though.

What is coming next:

  1. Comparison between libraries regarding e.g. speed, reliability, configurability, etc.
  2. More Snapshot testing samples (e.g. ScrollViews, Material you + dynamic colors...)
  3. Screenshot tests with other libraries: Facebook, ndtp/Testify, without library...
  4. Running snapshot tests on multiple devices/JVM in parallel
  5. Tips to remove flakiness
  6. Tips to increase test execution speed and more...

Code attribution

Special thanks to Alex Zhukovich for his CoffeeDrinksWithJetpackCompose projects, from which I've borrowed the Jetpack Compose examples!

Attribution of icons in the app

Icons made by Freepik from www.flaticon.com
Icons made by surang from www.flaticon.com