Android: https://github.com/MarathonLabs/marathon/tree/0.8.1/sample/android-app iOS: https://github.com/MarathonLabs/marathon/tree/0.8.1/sample/ios-app
Version 0.8.1 https://github.com/MarathonLabs/marathon/releases/tag/0.8.1
- Homebrew
- Custom
- Container
Input:
- Test and application binaries
- Marathonfile
Steps:
- Parse tests
- Map-reduce to available hardware
- Collect results
Output:
- Raw
- JUnit XML
- Allure results
- Runtime bill
- Files from devices
- Logs
- Screenshots/videos
- Visual
- html
- timeline
- allure html report (generated)
- Stdout
- Exit code
Why analytics TSDB is required?
- smart retries
- test balancing
- statistics for development teams
- InfluxDB v1 + v2
- Graphite
- Disabled a. Test probability of passing 0.0 b. Test expected duration 300s
Classification of retries
- Preventive (flakiness strategy)
- Post-factum (retry strategy)
- Uncompleted (uncompleted retry quota)
- Parallelisation (flakiness strategy)
- Launch overhead (batching strategy)
- Prediction error (sorting strategy)
-
Testing logic
-
Fully qualified test name
-
Fully qualified class name
-
Simple class name
-
Package name
-
Method name
-
Annotation/metaproperty name
Filtering by:
- Values
- Regex
- one more choice below
- Gitops via Marathonfile
- Dynamic configuration a. Envvar interpolation b. Filtering values file c. Custom templating (i.e. liquid)
Android debug bridge
- USB
- TCP/IP
a.
adb connect
b.adb pair
Usually only one is used, but some features such as gRPC access to emulator requires connecting to multiple adb servers instead of connecting devices via adb connect
- Local via dex-test-parser
Pros: Fast Cons: Unable to parse dynamically-generated tests
- Remote via
am instrument -e log true
Pros: Supports most dynamically-generated tests Cons: Slower, requires android device, custom testing frameworks might not properly implement the contract of am instrument i.e. flutter
Defaults to video recording up to 180sec
Screenshots via custom gif encoder
Priority: video -> screenshots -> none
Reduce the side-effects for each batch via pm clear
Note: resets the permissions
Internal implementation of marathon supports highly granular timeouts for each adb command
Use this feature to gain access to the device's external managements APIs:
- ADB
- Telnet (locally connected emulator only)
- gRPC (locally connected emulator only)
Samples:
- https://github.com/MarathonLabs/marathon/blob/0.8.1/sample/android-app/app/src/androidTest/java/com/example/AdbActivityTest.kt
- https://github.com/MarathonLabs/marathon/blob/0.8.1/sample/android-app/app/src/androidTest/java/com/example/ConsoleActivityTest.kt
- https://github.com/MarathonLabs/marathon/blob/0.8.1/sample/android-app/app/src/androidTest/java/com/example/GrpcActivityTest.kt
gRPC reference: https://github.com/Malinskiy/adam/blob/0.5.0/adam/src/main/proto/emulator_controller.proto
Allows tests to attach screenshots during execution to the Allure report
Marathon can push files before each batch and pull files after each batch of tests.
Supported locations:
- EXTERNAL_STORAGE
- APP_DATA (pull only)
- LOCAL_TMP
Aggregation modes:
- DEVICE
- POOL
- DEVICE_AND_POOL
- TEST_RUN
Apple doesn't support any notion of remote hardware. Marathon supports once-before test run provisioning via Marathondevices file with support for using:
- Local simulators
- Remote simulators via Secure Shell (ssh)
- Pre-created simulator by UDID (Unique Device Identifier)
- Simulator profile for creation/reuse (device profile + runtime)
Pull of xcresult files is supported
Note: no merging
Defaults to video recording
Screenshots via custom gif encoder
Priority: video -> screenshots -> none
Video:
- Codec: h264/hevc
- Mask: black/ignored
- Display: internal/external
Screenshots:
- Mask: black/ignored
- Display: internal/external
- Downscaling resolution
- Screenshot interval delay
Limits the side-effects via restart/clean
Reduces resource consumption via shutting down unused simulators
Caveat: need to specify bundle id for target app
- all - Apply the action to all services
- calendar - Allow access to calendar
- contacts-limited - Allow access to basic contact info
- contacts - Allow access to full contact details
- location - Allow access to location services when app is in use
- location-always - Allow access to location services at all times
- photos-add - Allow adding photos to the photo library
- photos - Allow full access to the photo library
- media-library - Allow access to the media library
- microphone - Allow access to audio input
- motion - Allow access to motion and fitness data
- reminders - Allow access to reminders
- siri - Allow use of the app with Siri
Granular operation timeouts are supported
Passing external parameters to tests, i.e. location of a mock server, or api environment to use for e2e testing
Batching + sorting by execution time + limited retries Android: no pm clear (very expensive)
Batching size 1 + sorting by execution time + lots of post-factum and uncompleted retries
Flakiness + Batching + sorting by execution time + reasonable retries
Map reduce without a single point of coordination: fragmentation filter