We publish the wTest+ tool and the evaluation results in this page.
- Open-source apps (download)
The following table lists the 38 open-source Android apps used in our evaluation.
Index | Package | Index | Package |
---|---|---|---|
1 | app.simple.inure | 20 | de.taz.android.app.free |
2 | ch.rmy.android.http_shortcuts.debug | 21 | fr.free.nrw.commons |
3 | com.activitymanager | 22 | im.fdx.v2ex.debug |
4 | com.boardgamegeek | 23 | io.github.wulkanowy |
5 | com.dar.nclientv2.debug | 24 | ml.docilealligator.infinityforreddit |
6 | com.ds.avare | 25 | name.boyle.chris.sgtpuzzles |
7 | com.flxrs.dankchat | 26 | net.bible.android.activity |
8 | com.gh4a | 27 | net.osmand.plus |
9 | com.github.k1rakishou.chan | 28 | nl.asymmetrics.droidshows |
10 | com.ichi2.anki | 29 | org.andstatus.app |
11 | com.kidozh.discuzhub | 30 | org.catrobat.paintroid |
12 | com.manimarank.spell4wiki | 31 | org.koitharu.kotatsu |
13 | com.nextgis.mobile | 32 | org.openhab.habdroid |
14 | com.nutomic.syncthingandroid | 33 | org.quantumbadger.redreader |
15 | de.blau.android | 34 | org.runnerup.free |
16 | de.danoeh.antennapod | 35 | org.wikipedia.alpha |
17 | de.freehamburger.debug | 36 | org.woheller69.omweather |
18 | de.herrmann_engel.rbv | 37 | org.woheller69.weather |
19 | de.tap.easy_xkcd | 38 | xyz.zedler.patrick.grocy |
- WebView-specific property coverage, WebView API call site coverage, and code coverage for open-source apps (link)
- bug1 (link)
- bug2 (link)
- bug3 (link)
- bug4 (link)
- bug5 (link)
- bug6 (link)
- bug7 (link)
- bug8 (link)
- bug9 (link)
- bug10 (link)
- bug11 (link)
- bug12 (link)
- bug13 (link)
- bug14 (link)
- bug15 (link)
- bug16 (link)
- bug17 (link)
- bug18 (link)
- bug19 (link)
- bug20 (link)
- bug21 (link)
- bug22 (link)
- bug23 (reported by email. WebViewActivity in de.taz.android.app.free) bug23.pdf
- bug24 (DataPolicyActivity in de.taz.android.app.free) bug24.pdf
- bug25 (IssueViewerActivity in de.taz.android.app.free) bug25.pdf
- bug26 (SearchActivity in de.taz.android.app.free) bug26.pdf
- bug27 (BookmarkViewerActivity in de.taz.android.app.free) bug27.pdf
- bug28 (PreferenceActivity in app.simple.inure) bug28.pdf
Our tool is composed of 3 parts. The first part staticly builds the GUI component transition graph. The second part instruments an app. The third part generates tests for the instrumented app. But before testing an app, a customized Android OS needs to be loaded by Android emulators or real devices.
The instrumented app needs to run on a customized Android system because we add one more ID field in java.lang.Object
in order to ease the variable tracking. It is very simple to use our customized Android system image, you just need to follow the following 2 steps
- Download our customized system.img, ramdisk.img, VerifiedBootParams.textproto (They are for Android 10)
- Replace the original
system.img
,ramdisk.img
,VerifiedBootParams.textproto
in your Android SDK with the downloaded files
Usually, these files are placed under <Android_SDK_Root>/system-images/android-29/google_apis/x86
Once you have finished the above two steps, you can try to start the Android emulator as usual.
Static analysis (download)
- Java 1.8 or above
- Create a folder for an app under test. The folder name should be the app's package name (We have created a folder for
org.woheller69.weather
under<path_to_GraphAndInstrumentation>/apps
). - Run
sh graph.sh APP_FOLDER APP_PACKAGE_NAME NON_LIB_PKGS
to statically build the GUI component transition graph for an app.NON_LIB_PKGS
is an array of package names separated by "," that are used to filter out library code in an app (e.g., if an app's application code is ina.b.c
anda.b.d
,NON_LIB_PKGS
should bea.b.c,a.b.d
). An example command can besh graph.sh apps/org.woheller69.weather org.woheller69.weather org.woheller69.weather
).
Instrumentation (packaged together with static analysis)
- Java 1.8 or above
- Python 2 or 3 (We have tested on Python 2.7.16 and Python 3.6.8)
- Node.js (We have tested on v16.13.2, v12.16.2, and v10.16.0. You may find these versions here. If you are using a Linux-based OS, you may install node by following the steps here)
- Run
sh instrument.sh APP_FOLDER APP_PACKAGE_NAME PORT_NUM NON_LIB_PKGS
to instrument an app.PORT_NUM
is a port used by the instrumented app to communicate with a nodejs server to instrument dynamically-loaded JavaScript code. It is recommended to be set as 3016, 3018, 3020, ..., etc. An example command can besh instrument.sh apps/org.woheller69.weather org.woheller69.weather 3016 org.woheller69.weather
). - When instrumentation finishes, you can see an
output
folder under theAPP_FOLDER
. The apk file whose name ends with-aligned-debugSigned.apk
is the instrumented apk.
Test generation (wTest+) (download)
export JAVA_HOME=YOUR_JAVA_HOME
(example:export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/
)export ANDROID_HOME=YOUR_ANDROID_HOME
(example:export ANDROID_HOME=/Library/Android/sdk
)export PATH=$PATH:${ANDROID_HOME}
export PATH=$PATH:<path_to_wTestPlus>/node_modules/.bin
(make sure you have downloadedwTestPlus
)export PATH=$PATH:${ANDROID_HOME}/emulator
export PATH=$PATH:${ANDROID_HOME}/platform-tools
- build-tools 27.0.3 (for ComboDroid) and 29.0.3 are required (they are usually at
${ANDROID_HOME}/build-tools
) - python3
- droidbot (link)
- Enter
<path_to_GraphAndInstrumentation>/js
and runsh launchServer.sh PORT_NUM
. This nodejs server is registered on aPORT_NUM
in order to recevice JavaScript code that is dynamically constructed from the app under test. The server is responsible for instrumenting the js code and sending the instrumented code back to the app.PORT_NUM
should be the same as the one used when instrumenting the app. An example command can besh launchServer.sh 3016
- Launch Appium by
appium -p APPIUM_PORT
(example:appium -p 4723
) - Launch an Android emulator
- Enter
<path_to_wTestPlus>
and runsh test.sh STRATEGY PATH_TO_APP_FOLDER TIME_LIMIT ANDROID_SDK_PATH EMULATOR_ID APPIUM_PORT AVD_NAME BUILD_TOOLS_VERSION SYSTEM_PORT PYTHON3 DROIDBOT
.STRATEGY
can bewVar_sa
(stands for wTest+), orwVar
(stands for wTest), orwDroid
, orapi
(stands for wTest-API), orcode
(stands for wTest-Code), orQTesting
, orComboDroid
, orFastbot
(stands for Fastbot2), orAdaT
.AVD_NAME
is the name of the android emulator.BUILD_TOOLS_VERSION
must be set as 29.0.3.SYSTEM_PORT
is a port used by appium if you test multiple apps in parallel. It is recommended to be set as 8200, 8201, 8202, etc.PYTHON3
is the path to your python3 command (you can check bywhich python3
).DROIDBOT
is the path to your droidbot command (you can check bywhich droidbot
). An example command can besh test.sh wVar_sa <path_to_GraphAndInstrumentation>/apps/org.woheller69.weather 3600 ${ANDROID_HOME} emulator-5554 4723 emulator0 29.0.3 8200 /usr/bin/python3 <root>/.local/bin/droidbot
. - When reaching the time limit, you can see the output under
<path_to_wTestPlus>/output/<STRATEGY>/<APP_PACKAGE_NAME>
(e.g.,<path_to_wTestPlus>/output/wVar_sa/org.woheller69.weather
). Under this folder,coverage.txt
contains the covered WebView-specific properties and WebView API call sites over 60 minutes (time0, time1, ..., time59 means 1 minute, 2 minute, ..., 60 minute). Code coverage is also reported.