mobile-dev-inc/maestro

Assertions on HTML (webview) not working

testcenter opened this issue · 0 comments

Is there an existing issue for this?

  • I have searched the existing issues and didn't find mine.

Steps to reproduce

The problem is related to maestro tests on Android.
The Bitmovin player uses a Webview with HTML/Javascript to visualize the video player UI elements.

  • Clone the Bitmovin Samples Repo
  • Build and install the BasicPlaybackKotlin example app
    • You need to provide a key in the AndroidManifest which can be retrieved via creating a free account on the Bitmovin page
  • Execute following config and flow via maestro
  • the assertion - assertVisible: "en" fails
    • Asserting on a physical device works as expected
    • Asserting on an emulator with google_apis_playstore works as expected
    • Asserting on an emulator with google_apis does not work
    • Asserting on maestro cloud fails. I assume, maestro cloud uses maestro start-device to create a device. It uses google_apis, see DeviceConfig

config.yaml

flows:
  - "DisplaysSubtitles.yaml"

DisplaysSubtitles.yaml

appId: com.bitmovin.player.samples.playback.basic
name: "Playback: Displays subtitles"
---
- launchApp
# toggle settings
- tapOn:
    text: "settings"
    retryTapIfNoChange: false
- extendedWaitUntil:
    visible: "subtitles"
    timeout: 10000
# open subtitle menu
- tapOn: "Select subtitle"
# select subtitle "en"
- tapOn: "en"
# !!! following assertion fails !!!
- assertVisible: "en"
# close settings menu
- tapOn:
    text: "settings"
    retryTapIfNoChange: false
# start playing
- tapOn:
    text: "Play/Pause"
    retryTapIfNoChange: false
# subtitle must be visible
- assertVisible: "Welcome to bitmovin's bitdash!"
- stopApp

Actual results

The assertion - assertVisible: "en" fails although visible. (inside a WebView)
Using google_apis_playstore instead of google_apis would work

Expected results

Assertions behave the same on google_apis and google_apis_playstore (might be a general issue when parsing WebView hierarchy of google_apis vs google_apis_playstore)

About app

  • Android App
  • open source sample app (link provided in the steps to reproduce)
  • native app
  • native framework (android views, xml)
  • WebView (html + javascript) is used to visualize the player UI elements

About environment

java -version

openjdk version "17.0.12" 2024-07-16
OpenJDK Runtime Environment (build 17.0.12+7-Ubuntu-1ubuntu222.04)
OpenJDK 64-Bit Server VM (build 17.0.12+7-Ubuntu-1ubuntu222.04, mixed mode, sharing)

uname -a

Linux ubuntu 5.15.0-119-generic #129-Ubuntu SMP Fri Aug 2 19:25:20 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

please note, the problem also happens on maestro cloud!

Logs

Logs
Waiting for flows to complete...

[Failed] Playback: Displays subtitles (35s) (Assertion is false: "en" is visible)

1/1 Flow Failed

Maestro version

1.38.1

How did you install Maestro?

install script (https://get.maestro.mobile.dev)

Anything else?

No response