Automatic screenshot generation for Marketing & Tests

Question

Automatic screenshot generation for Marketing & Tests

M123-dev opened this issue 4 years ago · 38 comments

M123-dev commented 4 years ago

What

We have around 80 languages in which we have products.
We have around 5 screenshots to make
A screenshot might need up to 10 scanned products.
The time to produce local feeling screenshots that are up to date with the current UI is tremendous.
And yet: “a majority of potential users will not read through the whole app description to find out, what your app is about. 50% of users make their purchase decisions based on the first impression”

Tasks

Write UI test to create the screenshots

Beta Give feedback

Write UI test to create a screenshot of scan mode #1447

P5 scan card 📢 Screenshot generation 🤳🥫 Scan 🧪 Tests
Write UI test to create a screenshot of the product page #1449

P5 📢 Screenshot generation 🧪 Tests
Write UI test to create a screenshot of history #1446

P5 history 📢 Screenshot generation 🧪 Tests
Write UI test to create a screenshot of personal search settings #1448

P5 ⚙️ Settings 📢 Screenshot generation 🧪 Tests
Add a step to Screenshot generation UI tests to clean the top bar #1596

📢 Screenshot generation
Add screenshot generation to GitHub Actions pipeline #1433

P5 📢 Screenshot generation
Options

Other tasks

Automate adding frames and slogans around screenshots
- fastlane frameit
Set up translation of slogans with Crowdin
#539

How

Design doc
Since Fastlane has no direct support for flutter screenshots, I think this could help.
https://medium.com/@nocnoc/automated-screenshots-for-flutter-f78be70cd5fd
https://github.com/mmcc007/screenshots
https://docs.fastlane.tools/getting-started/ios/screenshots/

Part of

#536

teolemon commented 3 years ago

Yes

Answer 1 · 2021-04-23T13:55:56.000Z

Judging from the issues in the repo, it seems the package is not maintained anymore, which is a pity

Answer 2 · 2021-04-23T15:01:06.000Z

@rolandgeider oh that's a shame, have you found an alternative for your project?

Answer 3 · 2021-04-23T15:11:05.000Z

sadly not, but I also haven't looked much into it. It's somehow possible to save the current screen contents to a file, but nothing so easy as that package, I was hoping you guys would have some ideas 😅

Answer 4 · 2021-04-23T15:15:33.000Z

@rolandgeider I have not yet dealt with the topic much more either, if I find a solution I will let you know.

Answer 5 · 2021-04-24T12:05:43.000Z

FYI, I have found this which looks promising, using only the flutter driver: https://medium.com/flutter-community/testing-flutter-ui-with-flutter-driver-c1583681e337

however it seems flutter_driver and flutter_launcher_icons have dependency incompatibilities 🙄 so I couldn't test it

Answer 6 · 2021-04-27T08:01:39.000Z

@rolandgeider I don't think we have a real alternative to the screenshot package.

Currently only a not merged fork supports null-safety. Its not future proof but its probably the best way to go for now.

Answer 7 · 2021-06-09T09:31:58.000Z

Someone suggested to transfer the package to the FlutterCommunity, hopefully it gets accepted.

fluttercommunity/community#74

Answer 8 · 2021-07-06T06:36:06.000Z

For the record, in another project I developed a shell that can change the language of the iOS simulator and run tests - which include screenshots.
Basically, on a mac book and not 100% automatic but I just have to start the right simulator and run the shell: all the screenshots of all languages will be created for that simulator. Then do it again for the other 2 screen configurations/simulators needed in the AppStore.

Answer 9 · 2021-07-22T11:24:20.000Z

Two days ago in the fellowship frontend workgroup sync @gspencergoog mentioned screenshot tests. That gave me the idea that we might be able to abuse a screenshot testing tool to generate screenshots for fastlane. I couldn't find the one he mentioned but golden_toolkit (by ebay) looks very promising.

A article with some additional information: https://medium.com/flutter-community/flutter-screenshot-testing-as-a-solid-ui-regression-tool-630221a621e4

Answer 10 · 2021-07-22T14:53:08.000Z

The one Flutter uses is Skia Gold. I don't think we can use their servers directly, unless we want to just use their code and set up our own instance that uses Google Cloud (it is open source, at least). I could ask the Skia team how hard/expensive it is to set up if you like.

Answer 11 · 2021-07-22T14:56:15.000Z

Anyway we need iOS screenshots (minimum is one in 3 different screen sizes in English), without that it's not possible to have it in the AppStore (maybe OK in TestFlight, though).

Answer 12 · 2021-07-22T15:14:43.000Z

@gspencergoog thanks for the offer, but I think it is easier and more uniform to do the process (at least for generations not for tests) with the goolden_toolkit and github actions.

We only have to think about if plain screenshots are to booring and if so how we edit them.

Answer 13 · 2021-07-22T15:16:06.000Z

Hey @M123-dev, you're supposed to be on vacation! ;)

Answer 14 · 2021-07-22T16:39:08.000Z

@M123-dev That makes sense.

Answer 15 · 2022-02-06T10:35:59.000Z

@monsieurtanuki note: I've managed to automate running the Screenshot tests suites on GitHub Actions for the classic apps, so we can run them on all platforms without blocking a personal computer

Answer 16 · 2022-02-06T10:40:56.000Z

@teolemon That means classic Android and classic iOS apps, but not flutter so far, right?

Answer 17 · 2022-02-06T11:24:40.000Z

Just did some further research and found another approach that doesn't use the not null safe screenshots package

https://blog.codemagic.io/flutter-automated-screenshot-testing/

Answer 18 · 2022-03-09T08:55:28.000Z

Is this an issue we can split for GSOC, asking newcomers to write one script each ?

Answer 19 · 2022-03-09T09:43:20.000Z

What do you mean with one script for each, one for iOS one for android?

I still think it's better to somehow do it in flutter but when there is no other way we could do it like that.

Another idea which just came to my mind. Running the app in debug mode in the browser, there we can switch the device and language and automate this with puppeteer. This probably won't run on github actions though

Answer 20 · 2022-03-09T09:50:48.000Z

@teolemon @M123-dev For my flutter/iOS app, months ago, I had to create a "test" that included screenshots (then I could use the same test for Android of course).
And I had to run it with unix scripts again and again, for different languages (env variable). Can't remember if I could restart automagically the iOS simulator or if I had to do it manually. Can check that.
Would that be compatible with github actions?
Is fastlane planning to include iOS screenshots one day?

Answer 21 · 2022-03-09T10:02:34.000Z

One script per scenario I meant @M123-dev . Same tech, but there are many screenshots to take
@monsieurtanuki We have already automated it for the classic apps, for both platforms (ios and android) using github actions, we should be able to scrap the .yml and the fastlane parts, the Flutter tests would need to be written

https://github.com/openfoodfacts/openfoodfacts-ios/blob/develop/.github/workflows/screenshots.yml
https://github.com/openfoodfacts/openfoodfacts-androidapp/pull/4206/files

Answer 22 · 2022-03-09T10:26:24.000Z

I guess the first big step would be to create the first test w/ screenshot and check that we're able to use that test to generate screenshots in tons of languages. And that the test is easy to copy/paste.
Then, perhaps split that test in several one-screenshot tests, though I think we could do everything in the same test session (at least that's what I did in my own project).

Answer 23 · 2022-03-14T10:32:29.000Z

@monsieurtanuki @M123-dev I've started the GitHub actions part in #1208

Answer 24 · 2022-03-14T10:41:40.000Z

@teolemon What do you need now? A test that takes one screenshot, and optionally a unix script that runs it?

Answer 25 · 2022-03-14T10:52:19.000Z

I'm not sure how the test should be written and how they should be run, but at the end of the day Github Actions can either run unix command, or run emulators (spawning an emulator is already part of my PR).
So no strong opinions.

Answer 26 · 2022-04-05T06:56:33.000Z

Two days ago in the fellowship frontend workgroup sync @gspencergoog mentioned screenshot tests. That gave me the idea that we might be able to abuse a screenshot testing tool to generate screenshots for fastlane. I couldn't find the one he mentioned but golden_toolkit (by ebay) looks very promising.

A article with some additional information: https://medium.com/flutter-community/flutter-screenshot-testing-as-a-solid-ui-regression-tool-630221a621e4

Has this package been tried and tested? Looks promising We can generate multiple Devices Screenshot along with Multiple parameterized screenshot for different languages

Answer 27 · 2022-04-05T07:06:02.000Z

@alphaNewrex I ran some tests with golden_tookit but they were a bit disappointing - good enough for tests, but not for proper screenshots:

Answer 28 · 2022-04-08T09:44:44.000Z

I've finally managed to create again screenshots on both Android and iOS. Done. Wasn't easy because of bugs and evolutions, but in the end it's not that hard.

The first (tricky) part is a fix of an flutter/iOS bug (cf. flutter/flutter#91668):

go to flutter/packages/integration_test/ios/Classes/IntegrationTestPlugin.m
edit method registerWithRegistrar this way

+ (void)registerWithRegistrar:(NSObject<FlutterPluginRegistrar> *)registrar {
    [[IntegrationTestPlugin instance] setupChannels:registrar.messenger];
}

then flutter clean

Then, let's create a new project (flutter create tanuki_project)

Edit pubspec.yaml:

dev_dependencies:
  integration_test: # add
    sdk: flutter    # add
  flutter_test:
    sdk: flutter

Create 2 folders at the same level as lib: test_driver and integration_test.

File test_driver/screenshot_driver.dart:

import 'dart:io';
import 'package:integration_test/integration_test_driver_extended.dart';

// cf. https://dev.to/mjablecnik/take-screenshot-during-flutter-integration-tests-435k
Future<void> main() async {
  try {
    await integrationDriver(
      onScreenshot: (String screenshotName, List<int> screenshotBytes) async {
        final File image = await File('screenshots/$screenshotName.png')
            .create(recursive: true);
        image.writeAsBytesSync(screenshotBytes);
        return true;
      },
    );
  } catch (e) {
    print('Error occured: $e');
  }
}

File integration_test/app_test.dart:

import 'dart:io';
import 'package:flutter/foundation.dart' show kIsWeb;
import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';

import 'package:tanuki_project/main.dart' as app;

Future<void> takeScreenshot(tester, binding, name) async {
  if ((!kIsWeb) && Platform.isAndroid) {
    await binding.convertFlutterSurfaceToImage();
    await tester.pumpAndSettle();
  }
  await binding.takeScreenshot(name);
}

void main() {
  final binding = IntegrationTestWidgetsFlutterBinding.ensureInitialized();

  group('end-to-end test', () {
    testWidgets('tap on the floating action button, verify counter',
        (WidgetTester tester) async {
      app.main();
      await tester.pumpAndSettle();

      // Verify the counter starts at 0.
      expect(find.text('0'), findsOneWidget);

      await takeScreenshot(tester, binding, 'test-screenshot-1');

      // Finds the floating action button to tap on.
      final Finder fab = find.byTooltip('Increment');

      // Emulate a tap on the floating action button.
      await tester.tap(fab);

      // Trigger a frame.
      await tester.pumpAndSettle();

      // Verify the counter increments by 1.
      expect(find.text('1'), findsOneWidget);
    });
  });
}

And the actual test run:

flutter drive --driver=test_driver/screenshot_driver.dart --target=integration_test/app_test.dart

The result here will be a file called test-screenshot-1.png in (new) folder screenshots.

Answer 29 · 2022-04-08T09:45:56.000Z

As a test, I'm going to create screenshots of the first (onboarding) page. We'll build up from there.

Answer 30 · 2022-04-08T17:50:58.000Z

That was painful, but I did it (iPhone 8 plus Simulator)!

Answer 31 · 2022-04-08T19:45:15.000Z

@teolemon I need your help: the cameras don't work on the emulator, and I have to replace them withpngs. Would you provide one for the screenshots? We'll see later which actual sizes we need.

Answer 32 · 2022-04-08T20:37:08.000Z

Answer 33 · 2022-04-09T06:02:22.000Z

Thank you @teolemon!
That said, as in smoothie there are many controls on the bottom part, we'd be better off with interesting data like barcodes on top of the image. Localizations may be an issue too. But that's a start.

Answer 34 · 2022-04-10T06:48:09.000Z

I could not test it before for Android, but here we are:

For the record, that confirms my low esteem for Stack and Position widgets.

Answer 35 · 2022-04-10T08:04:20.000Z

auto 407 / 670, for the record

Answer 36 · 2022-09-08T06:48:03.000Z

Mmm why I am assigned to this issue?

Answer 37 · 2022-09-08T08:33:06.000Z

unsure, fixing this