/TimeMachine

Time-travel Testing

Primary LanguagePythonGNU Lesser General Public License v3.0LGPL-3.0

TimeMachine

TimeMachine is an automated testing tool for Android apps, which can automatically jump to the most progressive state observed in the past when progress is slow.

TimeMachine leverages virtualization technology such as emulator to save an app state and restore it when needed. TimeMachine identifies an app state with GUI layout and memories all discovered states. When a state is considered interesting, e.g., new code is covered, TimeMachine saves the state. Meanwhile, TimeMachine observes most-recently-visited states to check whether progress is slow, e.g., being stuck in a state loop. Once progress is considered slow, TimeMachine restores the most progressive one from saved states for further exploration so that more program behavior is exercised in a short time.

Publication

The paper PDF can be found at https://zhendong2050.github.io/res/time-travel-testing-21-01-2020.pdf

@InProceedings{zhendong:icse:2020,
author = {Dong, Zhen and B{\"o}hme, Marcel and Cojocaru, Lucia and Roychoudhury, Abhik},
title = {Time-travel Testing of Android Apps},
booktitle = {Proceedings of the 42nd International Conference on Software Engineering},
series = {ICSE '20},
year = {2020},
pages={1-12}}

What is new?

For easy-to-use, we upgrade TimeMachine with following features:

  • Using the Android Emulator instead of Virtualbox
  • Using Jacoco for instrumenting apps instead of EMMA
  • Solving other issues such as adb connection issues during fuzzing

Prerequisites

  • Ububntu 18.04 64-bit or Mac-OSX 10.15
  • Android SDK with API 25 (ensuring adb, aapt, avdmanager, emulator correctly configured)
  • Python 2.7 (ensuring enum and uiautomator packages are installed)

Run TimeMachine

1. Clone Repos

  • Clone TimeMachine
# creating workspace
mkdir workspace
cd workspace

git clone https://github.com/DroidTest/TimeMachine.git

# creating dir for AUT
mkdir appTest
cd appTest

git clone --branch v3.4.2 https://github.com/TeamAmaze/AmazeFileManager.git

2. Instrument the app with Jacoco

  • Build an instrumented apk
# Add the jacoco plugin
echo -e "\napply plugin: 'jacoco'" >> AmazeFileManager/app/build.gradle
sed -i "`sed -n -e "/debug {/=" AmazeFileManager/app/build.gradle` a testCoverageEnabled true" AmazeFileManager/app/build.gradle

cp -r ../TimeMachine/JacocoIntegration/JacocoInstrument AmazeFileManager/app/src/main/java/com/amaze/filemanager

# Add package names
sed -i '1i package com.amaze.filemanager.JacocoInstrument;' AmazeFileManager/app/src/main/java/com/amaze/filemanager/JacocoInstrument/FinishListener.java
sed -i '1i package com.amaze.filemanager.JacocoInstrument;' AmazeFileManager/app/src/main/java/com/amaze/filemanager/JacocoInstrument/JacocoInstrumentation.java
sed -i '1i package com.amaze.filemanager.JacocoInstrument;' AmazeFileManager/app/src/main/java/com/amaze/filemanager/JacocoInstrument/SMSInstrumentedReceiver.java

# Register the BroadcastReceiver in AndroidManifest.xml
sed -i "`sed -n -e "/<\/application>/=" AmazeFileManager/app/src/main/AndroidManifest.xml` i <receiver android:name=\".JacocoInstrument.SMSInstrumentedReceiver\"><intent-filter><action android:name=\"edu.gatech.m3.emma.COLLECT_COVERAGE\"/></intent-filter></receiver>" AmazeFileManager/app/src/main/AndroidManifest.xml

# Build app with gradle
cd AmazeFileManager

./gradlew clean
./gradlew --no-daemon assembleDebug
  • Setup the apk in the test folder
cp app/build/outputs/apk/fdroid/debug/app-fdroid-debug.apk ../AmazeFileManager.apk

# Generate a class_files.json to describe the built directory
echo "{\"AmazeFileManager.apk\": {\"classfiles\": [\"AmazeFileManager/app/build/intermediates/javac/fdroidDebug/classes/\",\"AmazeFileManager/commons_compress_7z/build/intermediates/javac/debug/classes/\"]}}" > ../class_files.json

3. Check if the instrumented app works

# Launch emulator
sdkmanager "system-images;android-25;google_apis;x86"
avdmanager create avd -n avd0 -k "system-images;android-25;google_apis;x86" -d pixel_2_xl -c 1000M -f
nohup emulator -avd avd0 -writable-system &
adb devices
adb wait-for-device
adb root

# Run the apk on emulator 
adb install -g ../AmazeFileManager.apk
adb shell am start com.amaze.filemanager.debug/com.amaze.filemanager.activities.MainActivity

# Check if the coverage.ec file is generated. If so, the apk works well. 
adb shell am broadcast -a edu.gatech.m3.emma.COLLECT_COVERAGE
adb shell "cat /data/data/com.amaze.filemanager.debug/files/coverage.ec" 
adb emu kill

4. Test the apk with TimeMachine

  • Launch TimeMachine
cd ../../TimeMachine/fuzzingandroid

python2.7 main.py --avd avd0 --apk ../../appTest/AmazeFileManager.apk --time 1h -o ../../appTest/timemachine-results --no-headless
  • Check testing results
ls ../../appTest/timemachine-results/[output_dir_name]

# Expected results:
├── coverage.xml                current jacoco coverage report in xml
├── crashes.log                 crash logs in testing
├── data.csv                    coverage data in csv
├── ec_files                    dirs of *.ec generated by jacoco-agent
│   └── *.ec         
├── run_time.log                time of starting test
└── timemachine-run.log         runtime log of timemachine