sergio-sastre/ComposablePreviewScanner

A library to help auto-generate screenshot tests from Composable Previews with any screenshot testing library: JVM-based (i.e. Paparazzi, Roborazzi) as well as Instrumentation-based (i.e. Shot, Dropshots, Android-Testify, etc.)

Composable Preview Scanner

A library to help auto-generate screenshot tests from Composable Previews with any screenshot testing library: JVM-based (i.e. Paparazzi, Roborazzi) as well as Instrumentation-based (i.e. Shot, Dropshots, Android-Testify, etc.)

Comparison with other solutions

	Composable Preview Scanner	Showkase	Compose Preview Screenshot Testing
Independent of AGP version	✅	✅	❌
Library-agnostic solution	✅	✅	❌1
Scans previews in "main" source	✅	✅	❌2
Preview Infos available	✅	❌3	✅
Specific Config (e.g. for Libs) available	✅4	❌	⚠️5
Compose Multiplatform Previews support	*✅6	❌7	❌

¹ Compose Preview Screenshot Testing is a standalone solution based on LayoutLib, whereas ComposablePreviewScanner and Showkase provide Composables' infos so you can run screenshot tests with your favourite screenshot testing library.

² Compose Preview Screenshot Testing requires to put the previews in a brand-new "screenshotTest" source. ComposablePreviewScanner and Showkase only work with previews in the "main" source, so if you want to have the previews used for screenshot tests separate, I recommend to create an extra module for them and their screenshot tests and do not include that module in :app.

³ Showkase components only hold information about the Composable, but not about the Preview Info (i.e. ApiLevel, Locale, UiMode, FontScale...).

⁴ ComposablePreviewScanner supports adding extra lib-config (e.g. Paparazzi's Rendering Mode or Roborazzi's compare options) in the form of annotations that are additionally added to the preview. You can check how in the examples below in Jvm Screenshot Tests and Instrumentation Screenshot Tests respectively.

⁵ Compose Preview Screenshot Testing supports only general tolerance via gradle plugin from version 0.0.1-alpha06

⁶ ComposablePreviewScanner provides some Compose Multiplatform support: It can scan @Previews in common, as described in the Compose-Multiplatform section. Compose-Desktop @Previews are still not supported out of the box because of this issue, but you can check the Compose-Desktop support section for a workaround.

⁷ Showkase: Compose Multiplatform Support

ComposablePreviewScanner also works with:

• @PreviewParameters
• Multi-Previews, including @PreviewScreenSizes, @PreviewFontScales, @PreviewLightDark, and @PreviewDynamicColors.
• private @Previews (from version 0.1.3+)
• @Previews inside public classes¹ (from version 0.3.0+), not nested classes though

but does not work with

• @Previews that are not located in the "main" source

¹ Compose Preview Screenshot Testing tool requires to put your @Previews inside a class. By supporting this, you can keep @Previews in a separate class, and keep that class in the "main" source to auto-generate screenshot tests from them with ComposablePreviewScanner and Roborazzi, Paparazzi or any instrumentation-based screenshot testing library, or move that class to "screenshotTest" source to auto-generate screenshot tests from them with Compose Preview Screenshot Testing.

How to set up

Warning

Beware the prefixes:
Maven Central -> io.github
Jitpack -> com.github

Maven central (since 0.3.2)

dependencies {
   // jvm tests (e.g. with Roborazzi & Paparazzi)
   testImplementation("io.github.sergio-sastre.ComposablePreviewScanner:android:0.3.2")

   // instrumentation tests (e.g. with Shot, Dropshots & Android-Testify)
   debugImplementation("io.github.sergio-sastre.ComposablePreviewScanner:android:0.3.2")
   
   // compose multiplatform (jvm-targets)
   testImplementation("io.github.sergio-sastre.ComposablePreviewScanner:jvm:0.3.2")
}

Jitpack

Add jitpack to your root build.gradle file:

allprojects {
   repositories {
      maven { url 'https://jitpack.io' }
   }
}

dependencies {
   // jvm tests (e.g. with Roborazzi & Paparazzi)
   testImplementation("com.github.sergio-sastre.ComposablePreviewScanner:android:<version>")

   // instrumentation tests (e.g. with Shot, Dropshots & Android-Testify)
   debugImplementation("com.github.sergio-sastre.ComposablePreviewScanner:android:<version>")
   
   // compose multiplatform (jvm-targets), from 0.2.0
   testImplementation("com.github.sergio-sastre.ComposablePreviewScanner:jvm:<version>")
}

How to use

Direct links

1. Jvm Screenshot Tests
   1.1 Paparazzi
   1.2 Roborazzi
   
2. Instrumentation Screenshot Tests
3. Compose Multiplatform Support

API

The API is pretty simple:

AndroidComposablePreviewScanner() // or CommonComposablePreviewScanner(), see Compose Multiplatform section
    .scanPackageTrees(
        include = listOf("your.package", "your.package2"),
        exclude = listOf("your.package.subpackage1", "your.package2.subpackage1")
    )
    // options to filter scanned previews
    .excludeIfAnnotatedWithAnyOf(
        ExcludeForScreenshot::class.java, 
        ExcludeForScreenshot2::class.java
    )
    .includeAnnotationInfoForAllOf(
        ScreenshotConfig::class.java,
        ScreenshotConfig2::class.java
    )
    .includePrivatePreviews() // Otherwise they are ignored
    .filterPreviews { 
        // filter by any previewInfo: name, group, apiLevel, locale, uiMode, fontScale...
        previewInfo ->  previewInfo.apiLevel == 30 
    }
    // ---
    .getPreviews()

There are 2 more options to scan previews:

1. All Packages: scanAllPackages(). This might require a huge amount of memory since it would scan not only in a set of packages, but in all packages used in your app/module (i.e. also in its transitive dependencies). This is in 99% of the cases unnecessary, and scanning the main package trees of your module should be sufficient.
2. From a file containing the ScanResult. This speeds up your screenshot tests, since avoids the time-consuming process of scanning each time by reusing the previously scanned data:
   2.1. scanFile(jsonFile: File). Use this for JVM-based screenshot testing libraries (i.e. Roborazzi & Paparazzi).
   2.2. scanFile(inputStream: InputStream). This is useful for Instrumentation-based screenshot testing libraries.
   
   To dump the ScanResult into a File, the easiest is to create a unit test for that. Likely to be provided via gradle task in the future:

class SaveScanResultInFiles {
    @Test
    fun `task -- save scan result in file`() {
        val scanResultFileName = "scan_result.json"

        ScanResultDumper()
            .scanPackageTrees("my.package")
            // for unit tests
            .dumpScanResultToFile(scanResultFileName)
            // for instrumentation tests
            .dumpScanResultToFileInAssets(
               flavourName = "myFlavour",
               fileName = scanResultFileName
            )
    }
}

Warning

You'll have to ensure that the file is already generated before running the screenshot tests. You'll also need to keep it up to date.

JVM Screenshot Tests

Paparazzi

You can find executable examples here

Let's say we want to enable some custom Paparazzi Config for some Previews, for instance a maxPercentDifferent value

1. Define your own annotation for the Lib config.

annotation class PaparazziConfig(val maxPercentDifferent: Double)

2. Annotate the corresponding Previews accordingly (you do not need to annotate all):

@PaparazziConfig(maxPercentDifferent = 0.1F)
@Preview(uiMode = Configuration.UI_MODE_NIGHT_YES)
@Composable
fun MyComposable(){
    ...
}

3. Include your annotation info in the Preview

object ComposablePreviewProvider : TestParameter.TestParameterValuesProvider {
    override fun provideValues(): List<ComposablePreview<AndroidPreviewInfo>> =
        AndroidComposablePreviewScanner()
            .scanPackageTrees("my.package", "my.package2")
            .includeAnnotationInfoForAllOf(PaparazziConfig::class.java)
            ... // any other filtering option
            .getPreviews()
}

4. Map the PreviewInfo and PaparazziConfig values. For instance, you can use a custom class for that.

object PaparazziPreviewRule {
    fun createFor(preview: ComposablePreview<AndroidPreviewInfo>): Paparazzi =
        Paparazzi(
            deviceConfig = PIXEL_4A.copy(
                nightMode =
                   when(preview.previewInfo.uiMode and Configuration.UI_MODE_NIGHT_MASK == Configuration.UI_MODE_NIGHT_YES) {
                      true -> NightMode.NIGHT
                      false -> NightMode.NOTNIGHT
                }
                ... // other configurations
            ),
            maxPercentDifference = preview.getAnnotation<PaparazziConfig>()?.maxPercentDifference ?: 0F
        )
}

5. Create the corresponding Parameterized Test:

@RunWith(TestParameterInjector::class)
class PreviewTestParameterTests(
    @TestParameter(valuesProvider = ComposablePreviewProvider::class)
    val preview: ComposablePreview<AndroidPreviewInfo>,
) {

    @get:Rule
    val paparazzi: Paparazzi = PaparazziPreviewRule.createFor(preview)

    @Test
    fun snapshot() {
        paparazzi.snapshot {
            preview()
        }
    }
}

6. Run these Paparazzi tests together with the existing ones by executing the corresponding command e.g. ./gradlew yourModule:recordPaparazziDebug

Roborazzi

You can find executable examples here

Let's say we want to enable some custom Roborazzi Config for some Previews, for instance a maxPercentDifferent value

1. Define your own annotation for the Lib Config.

annotation class RoborazziConfig(val comparisonThreshold: Double)

2. Annotate the corresponding Previews accordingly:

@RoborazziConfig(comparisonThreshold = 0.1)
@Preview(uiMode = Configuration.UI_MODE_NIGHT_YES)
@Composable
fun MyComposable(){
    ...
}

3. Map the PreviewInfo and RoborazziConfig values. For instance, you can use a custom class for that.

object RoborazziOptionsMapper {
    fun createFor(preview: ComposablePreview<AndroidPreviewInfo>): RoborazziOptions =
       preview.getAnnotation<RoborazziConfig>()?.let { config ->
          RoborazziOptions(
             compareOptions = CompareOptions(resultValidator = ThresholdValidator(config.comparisonThreshold))
          )
       } ?: RoborazziOptions()
}

object RobolectricPreviewInfosApplier {
    fun applyFor(preview: ComposablePreview<AndroidPreviewInfo>) {
        val uiMode = nightMode =
           when(preview.previewInfo.uiMode and Configuration.UI_MODE_NIGHT_MASK == Configuration.UI_MODE_NIGHT_YES) {
              true -> "+night"
              false -> "+notnight"
           }
        RuntimeEnvironment.setQualifiers(uiMode)
       ... // other configurations
    }
}

Check the following link for a full list of Robolectric device qualifiers and this blog post on how to set the cumulative Qualifiers dynamically

4. Create the corresponding Parameterized Test:

@RunWith(ParameterizedRobolectricTestRunner::class)
class PreviewParameterizedTests(
    private val preview: ComposablePreview<AndroidPreviewInfo>,
) {

    companion object {
        // Optimization: This avoids scanning for every test
        private val cachedPreviews: List<ComposablePreview<AndroidPreviewInfo>> by lazy {
            AndroidComposablePreviewScanner()
                .scanPackageTrees("your.package", "your.package2")
                .filterPreviews { previewParams -> previewParams.apiLevel == 30 }
                .includeAnnotationInfoForAllOf(RoborazziConfig::class.java)
                .getPreviews()
        }

        @JvmStatic
        @ParameterizedRobolectricTestRunner.Parameters
        fun values(): List<ComposablePreview<AndroidPreviewInfo>> = cachedPreviews
    }

    @GraphicsMode(GraphicsMode.Mode.NATIVE)
    @Config(sdk = [30]) // same as the previews we've filtered
    @Test
    fun snapshot() {
        RobolectricPreviewInfosApplier.applyFor(preview)
        
        captureRoboImage(
           roborazziOptions = RoborazziOptionsMapper.createFor(preview)
        ) {
            preview()
        }
    }
}

5. Run these Roborazzi tests together with the existing ones by executing the corresponding command e.g. ./gradlew yourModule:recordRoborazziDebug

Instrumentation Screenshot Tests

Android does not use the standard Java bytecode format and does not actually even have a runtime classpath. Therefore, the simplest way to support instrumentation tests, is to...

1. run the scan in a unit test & save it in a file accessible by instrumentation tests e.g. in assets

class SaveScanResultInAssets {
    @Test
    fun `task -- save scan result in assets`() {
        val scanResultFileName = "scan_result.json"

        ScanResultDumper()
            .scanPackageTrees("my.package")
            .dumpScanResultToFileInAssets(
                fileName = scanResultFileName
            )

        assert(
           assetsFilePath(
               fileName = scanResultFileName
           ).exists())
    }
}

Ensure that the .json with the scan result is up-to-date before executing the instrumentation screenshot tests. For instance, execute that test always before your instrumentation screenshot tests. Ideally, this scanning could be done via a Gradle Plugin in the future instead of by running it in a unit test.

2. Now proceed to prepare your Composable Preview Tests with, for instance, Dropshots. Let's say we want to enable some custom Dropshots Config for some Previews, for instance a maxPercentDifferent value.
   • Define your own annotation

     annotation class DropshotsConfig(val comparisonThreshold: Double)

   • Annotate the corresponding Previews accordingly:

   @DropshotsConfig(comparisonThreshold = 0.15)
   @Preview(uiMode = Configuration.UI_MODE_NIGHT_YES)
   @Composable
   fun MyComposable(){
      ...
   }

   • Map the PreviewInfo and DropshotsConfig values. For instance, you can use a custom class for that. To map the Preview Info values, I recommend to use the ActivityScenarioForComposableRule of AndroidUiTestingUtils

   object DropshotsPreviewRule {
      fun createFor(preview: ComposablePreview<AndroidPreviewInfo>): Dropshots =
         preview.getAnnotation<DropshotsConfig>()?.let { config ->
            Dropshots(
               resultValidator = ThresholdValidator(config.comparisonThreshold))
            )
         } ?: Dropshots()
   }
   
   object ActivityScenarioForComposablePreviewRule {
      fun createFor(preview: ComposablePreview<AndroidPreviewInfo>): ActivityScenarioForComposableRule {
         val uiMode = when(preview.previewInfo.uiMode and Configuration.UI_MODE_NIGHT_MASK == Configuration.UI_MODE_NIGHT_YES) {
            true -> UiMode.NIGHT
            false -> UiMode.DAY
         }
         return ActivityScenarioForComposableRule(
            config = ComposableConfigItem(
                uiMode = uiMode
                ... // other configurations
            )
         )    
      }
   }

   • Create the corresponding Parameterized Test:

   @RunWith(ParameterizedTestRunner::class)
   class PreviewParameterizedTests(
      private val preview: ComposablePreview<AndroidPreviewInfo>,
   ) {
   
      companion object {
        private val cachedPreviews: List<ComposablePreview<AndroidPreviewInfo>> by lazy {
            AndroidComposablePreviewScanner()
                  .scanFile(getInstrumentation().context.assets.open("scan_result.json"))
                  .includeAnnotationInfoForAllOf(DropshotsConfig::class.java)
                  .getPreviews()
        }
   
         @JvmStatic
         @ParameterizedTestRunner.Parameters
         fun values(): List<AndroidComposablePreview<PreviewInfo>> = cachedPreviews   
      }
   
      @get:Rule
      val dropshots: Dropshots = DropshotsPreviewRule.createFor(preview)
   
      @get:Rule
      val activityScenarioForComposable: ActivityScenarioForComposableTestRule = 
            ActivityScenarioForComposablePreviewTestRule.createFor(preview)
   
      @Test
      fun snapshot() {
        activityScenarioForComposableRule.onActivity {
            it.setContent {
                preview()
            }
        }
   
        dropshots.assertSnapshot(
            view = activityScenarioForComposable.activity.waitForComposeView()
        )
      }
   }

   • Run these Dropshots tests together with the existing ones by executing the corresponding command e.g. ./gradlew yourModule:connectedAndroidTest -Pdropshots.record

Warning

Beware that Locale Strings in Preview Infos, unlike AndroidUiTestingUtils, use The BCP-47 tag but with + instead of - as separators, and have the prefix b+. Therefore, the BCP-47 tag "zh-Hans-CN" would be written as "b+zh+Hans+CN" instead. So for this case, you'd have to convert locale "b+zh+Hans+CN" to "zh-Hans-CN" in order to use it with AndroidUiTestingUtils

Advanced Usage

ComposablePreviewScanner also provides a class to customize the name of the generated screenshots based on its Preview Info. By default, it does not include the Preview Info in the screenshot file name if it is the same as its default value, but it can be configured to behave differently. That means, for @Preview(showBackground = false), showBackground would not be included in the screenshot file name since it is the default.

fun createScreenshotIdFor(preview: ComposablePreview<AndroidPreviewInfo>) = 
    AndroidPreviewScreenshotIdBuilder(preview)
        // Paparazzi screenshot names already include className and methodName
        // so ignore them to avoid them duplicated what might throw a FileNotFoundException
        // due to the longName
       .ignoreClassName()
       .ignoreMethodName()
            
       .ignoreForId("heightDp")
       .ignoreForId("widthDp")
       .overrideDefaultIdFor(
           previewInfoName = "showBackground",
           applyInfoValue = {
               when (it.showBackground) {
                   true -> "WITH_BACKGROUND"
                   false -> "WITHOUT_BACKGROUND"
               }
           }
       )
       .build()

and then in your test

@Test
fun snapshot() {
    paparazzi.snapshot(
        name = createScreenshotIdFor(preview)
    ) {
        preview()
    }
}

So, for the following Preview

class MyClass {
    
    @Preview(widthDp = 33, heightDp = 33, fontScale = 1.5f)
    @Composable
    fun MyComposable(){
        ...
    }
}

createScreenshotIdFor(preview) will generate the following id: "MyClass.MyComposable.FONT_1_5f_WITHOUT_BACKGROUND"

How It works

This library is written on top of ClassGraph, an uber-fast parallelized classpath scanner.

Classpath can scan everything that is available either at bytecode level or at runtime. This is also the case of annotations without retention or with either AnnotationRetention.BINARY or AnnotationRetention.RUNTIME, like Android Composable Previews

package androidx.compose.ui.tooling.preview
...
@Retention(AnnotationRetention.BINARY)
annotation class Preview(
...
)

However, those with AnnotationRetention.SOURCE are not visible to Classgraph. Such annotations are mainly used for IDE tooling, and that is the case for the Compose-Desktop Preview annotation.

package androidx.compose.desktop.ui.tooling.preview
...
@Retention(AnnotationRetention.SOURCE)
annotation class Preview

Compose Multiplatform Support

Since Compose Multiplatform 1.6.0, Jetbrains has added support for @Previews in common. ComposablePreviewScanner can also scan such Previews when running on any jvm-target, like

• Android
• Desktop
• Jvm

ComposablePreviewScanner provides a CommonComposablePreviewScanner for that purpose.

Assuming that you have:

• some Compose Multiplatform @Previews defined in common
• some Android screenshot tests in place

Here is how you could also run screenshot tests for those Compose Multiplatform @Previews together, for instance, with Roborazzi (would also work with Paparazzi or any Instrumentation-based library).

1. Add :jvm dependency from ComposablePreviewScanner 0.2.0+ testImplementation("com.github.sergio-sastre.ComposablePreviewScanner:jvm:<version>")

2. Add an additional Parameterized screenshot test for these Compose Multiplatform @Previews

@RunWith(ParameterizedRobolectricTestRunner::class)
class PreviewParameterizedTests(
    private val preview: ComposablePreview<CommonPreviewInfo>,
) {

    companion object {
        // Optimization: This avoids scanning for every test
        private val cachedPreviews: List<ComposablePreview<CommonPreviewInfo>> by lazy {
            CommonComposablePreviewScanner()
                .scanPackageTrees("your.package", "your.package2") // those where your common @Previews are located
                .includeAnnotationInfoForAllOf(RoborazziConfig::class.java)
                .getPreviews()
        }

        @JvmStatic
        @ParameterizedRobolectricTestRunner.Parameters
        fun values(): List<ComposablePreview<CommonPreviewInfo>> = cachedPreviews
    }

    @GraphicsMode(GraphicsMode.Mode.NATIVE)
    @Config(sdk = [30]) // same as the previews we've filtered
    @Test
    fun snapshot() {
        captureRoboImage(
           roborazziOptions = RoborazziOptionsMapper.createFor(preview)
        ) {
            preview()
        }
    }
}

3. Run these screenshot tests together with the existing ones by executing the corresponding command e.g. ./gradlew yourModule:recordRoborazziDebug

Compose-Desktop Support

As we've seen in the previous section How it works, Compose-Desktop previews are still not visible to ClassGraph since they use AnnotationRetention.SOURCE. There is already an open issue to change it to AnnotationRetention.BINARY, which would allow ClassGraph to find them.

In the meanwhile, it is also possible to workaround this limitation with ComposablePreviewScanner as follows.

1. Add :jvm dependency from ComposablePreviewScanner 0.2.0+ and use Roborazzi, since it is the only Screenshot Testing Library that supports Compose-Desktop testImplementation("com.github.sergio-sastre.ComposablePreviewScanner:jvm:<version>")

2. Configure Roborazzi as described in the corresponding "Multiplatform support" section

3. Define your own annotation. The only condition is not to define AnnotationRetention.SOURCE

   package my.package.path
      
   @Target(AnnotationTarget.FUNCTION)
   annotation class VisibleForScreenshotTesting

4. Annotate the Desktop Composables you want to generate screenshot tests for with this annotation, e.g.

   class MyClass {

        @VisibleForScreenshotTesting
        @Preview // It'd also work without this annotation
        @Composable
        fun MyDesktopComposable() { 
            ...
        }
   }

5. Create the test. Unfortunately, kotlin.test doesn't provide any means for parameterized tests, so we'll need to iterate through all previews. That has the disadvantage that, if it fails to record/verify one of the preview screenshots, subsequent screenshots won't be recorded/verified (hard asserts).

class DesktopPreviewScreenshotTests {

   @OptIn(ExperimentalTestApi::class)
   @Test
   fun desktopPreviews() {
      ROBORAZZI_DEBUG = true
      val listComposables = JvmAnnotationScanner(
         annotationToScanClassName = "my.package.path.VisibleForScreenshotTesting"
      )
         .scanPackageTrees("my.package.tree1", "my.package.tree2")
         .getPreviews()

      listComposables.forEach { composablePreview ->
         runDesktopComposeUiTest {
            setContent {
               composablePreview()
            }
            onRoot().captureRoboImage(
               filePath = "$composablePreview.png", // optional, works without it too
            )
         }
      }
   }
}

6. Run these Roborazzi tests by executing the corresponding command e.g. ./gradlew yourModule:recordRoborazziJvm (if using the Kotlin Jvm Plugin)

Tech talks

In these tech-talks have also been mentioned the benefits of using ComposablePreviewScanner:

• DroidKaigi 2024 [in JA 🇯🇵 with EN 🇬🇧 slides]:
  Understand the mechanism! Let's do screenshots testing of Compose Previews with various variations by Sumio Toyama

Composable Preview Scanner logo modified from one by Freepik - Flaticon