mobile-dev-inc/maestro

Flakiness often after animations between screens

Opened this issue · 2 comments

Describe the bug
We very frequently have tests flake/fail on button press/tap. The failures are random and can occur on almost any button press. Seems like the press is executed by maestro but doesn't register in the app since things are moving. Adding -waitForAnimationToEnd after every step slightly improves the success rate. The failures tend to happen more frequently after an animation/motion like navigating between React Navigation pages or when the button is on a swipeable action sheet. Not sure what the best way to resolve would be but perhaps the ability to add a per action "minimumWait" time before executing the action after the element is found.

For instance:

- tapOn:
  text: My Button
  minimumWait: 200

would wait an additional 200ms after My Button is found to execute the tap.

Another potential solution would be to retry an action if the UI doesnt change after executing the action. Or retry previous step when the next step fails.

To Reproduce

Have an app with lots of moving/animating parts and execute actions right after navigating pages or things move.

Expected behavior

Here is an example. In this example the Acknowledge button tap didn't execute despite being found and visible.

Screen Shot 2024-03-19 at 1 46 19 PM

This should work consistently without flaking.

Environment information (please complete the following information):

  • Maestro version: Maestro Cloud Default
  • Platform: iOS version 15.5
  • Framework: Expo
  • Device model and OS version: Maestro Cloud Default
  • Simulator, emulator or physical device: Maestro Cloud Default
  • Host: Maestro Cloud Default

is waitForAnimationToEnd useful here? Or will it never settle to something a bit more static?

It definitely has helped, but i believe mostly cause it introduces a slight additional delay which allows the UI to settle.
After investigating more it seems the main root cause is React Navigation animations which cause a high rate of flakiness post transition.By turning off animations in React Navigation we've had a huge improvement in failure rate