/thesis-mobile-consent-dialogs

Source code, data, presentation, and notes for my master’s thesis on consent dialogs in mobile apps on Android and iOS

Primary LanguageJupyter NotebookMIT LicenseMIT

Informed Consent? A Study of “Consent Dialogs” on Android and iOS

Source code, data, presentation, and notes for my master’s thesis on consent dialogs in mobile apps on Android and iOS

Consent dialogs have become ubiquitous with seemingly every website and app pleading users to agree to their personal data being processed and their behaviour being tracked, often with the help of tens or even hundreds of third-party companies. They are an effort by website and app publishers to comply with data protection legislation like the GDPR, which imposes strict limits on how companies can process data. Previous research has established that companies often apply dark patterns to illegally nudge users into agreeing and that at the same time tracking is more common than ever with both websites and apps regularly automatically transmitting telemetry data.

But so far, there has been almost no research into consent dialogs on mobile. For my master’s thesis, I studied consent dialogs on Android and iOS in an automated and dynamic manner, analysing 4,388 popular apps from both platforms. I identified different types of consent elements in the apps and analysed their prevalence. I also identified dark patterns and violations by the apps based on a list of criteria for a legally compliant consent dialog that I have compiled. Finally, I measured the effect of the user’s choice in the consent dialog by comparing the traffic from before any interaction with the traffic after accepting and rejecting the dialog and analysing contacted trackers and transmitted data types. The results show that more than 90 % of consent dialogs implement at least one dark pattern and that a majority of apps transmits tracking data regardless of consent status.

Setup

Preparation on macOS

  • Run brew install postgresql to be able to install psycopg2 later (ref).
  • Install libimobiledevice: brew install libimobiledevice ideviceinstaller
  • Install sshpass: brew install esolitos/ipa/sshpass
  • SSH into the iDevice manually once to add it to the list of known hosts.

Preparation for all systems

npm install -g appium appium-doctor
# Run `appium-doctor --android` or `appium-doctor --ios` and resolve the issues.

git clone https://github.com/baltpeter/thesis-mobile-consent-dialogs
cd thesis-mobile-consent-dialogs
yarn

python3 -m venv venv
source venv/bin/activate
pip install mitmproxy frida-tools objection python-dotenv psycopg2-binary

cd src
cp .env.sample .env
# Edit the values accordingly.
nano .env

Steps for setting up Appium under iOS

based on: https://appium.io/docs/en/drivers/ios-xcuitest-real-devices/ and https://makaka.org/unity-tutorials/test-ios-app-without-developer-account

  1. Create an Apple developer account (free is fine) and use that to log into XCode.
  2. Connect the iPhone via USB.
  3. Start XCode. In the menu bar, click Window and Devices and Simulators. Ensure the iPhone is available there.
  4. In a terminal, start appium.
  5. Follow the steps at https://appium.io/docs/en/drivers/ios-xcuitest-real-devices/#full-manual-configuration (full manual configuration).
  6. Open Appium Inspector and create a session with the following desired capabilities to test (adapted to your configuration):
{
  "appium:xcodeOrgId": "<org-id>",
  "appium:xcodeSigningId": "Apple Development",
  "appium:udid": "auto",
  "platformName": "iOS",
  "appium:app": "/path/to/app.ipa",
  "appium:automationName": "XCUITest",
  "appium:deviceName": "iPhone",
  "appium:updatedWDABundleId": "tld.your-domain.WebDriverAgentRunner"
}

Replace appium:app with the path to any IPA, appium:updatedWDABundleId with the ID chosen in the previous step, and appium:xcodeOrgId with the personal team ID. To find that, open the Keychain Access app. On the left, select login under Default Keychains, then choose My Certificates in the top bar. Doubleclick the correct certificate. The ID is listed under Organizational Unit. 6. Use the latter two values for running the analysis.

Device preparation

iOS

Android

  • Make sure to use an x86_64 emulator (Android 11).
  • Uninstall unnecessary Google apps to avoid their background traffic: 
    adb shell 'pm uninstall --user 0 com.android.chrome'
adb shell 'pm uninstall --user 0 com.google.android.apps.docs'
adb shell 'pm uninstall --user 0 com.google.android.apps.maps'
adb shell 'pm uninstall --user 0 com.google.android.apps.messaging'
adb shell 'pm uninstall --user 0 com.google.android.apps.photos'
adb shell 'pm uninstall --user 0 com.google.android.apps.pixelmigrate'
adb shell 'pm uninstall --user 0 com.google.android.apps.wellbeing'
adb shell 'pm uninstall --user 0 com.google.android.apps.youtube.music'
adb shell 'pm uninstall --user 0 com.google.android.gm'
adb shell 'pm uninstall --user 0 com.google.android.googlequicksearchbox'
adb shell 'pm uninstall --user 0 com.google.android.videos'
adb shell 'pm uninstall --user 0 com.google.android.youtube'
adb shell 'pm uninstall --user 0 com.google.mainline.telemetry'
  • Disable captive portal checks to further reduce background noise: 
    adb shell 'settings put global captive_portal_detection_enabled 0'
adb shell 'settings put global captive_portal_server localhost'
adb shell 'settings put global captive_portal_mode 0'
  • Install the mitmproxy certificate as root CA (emu needs to be started with -writable-system): 
    # Yields <hash>.
openssl x509 -inform PEM -subject_hash_old -in ~/.mitmproxy/mitmproxy-ca-cert.pem | head -1
cp ~/.mitmproxy/mitmproxy-ca-cert.pem <hash>.0

adb root
adb shell avbctl disable-verification
adb disable-verity
adb reboot
adb root
adb remount

adb push <hash>.0 /sdcard/
adb shell 'mv /sdcard/<hash>.0 /system/etc/security/cacerts/'
adb shell 'chmod 644 /system/etc/security/cacerts/<hash>.0'
adb reboot
  • Install Frida on the emulator (note that the version must match the one on the host): 
    wget https://github.com/frida/frida/releases/download/15.1.12/frida-server-15.1.12-android-x86_64.xz
7z x frida-server-15.1.12-android-x86_64.xz
adb push frida-server-15.1.12-android-x86_64 /data/local/tmp/frida-server
adb shell 'chmod 777 /data/local/tmp/frida-server'

adb root
adb shell 'nohup /data/local/tmp/frida-server >/dev/null 2>&1 &'
# Should have `frida-server`.
frida-ps -U | grep frida

Honey data

This is the honey data that I set in my analysis. You should probably change it. Remember to also change query.ts accordingly.

  • Contact: JGKfozntbF TBFFZbBYea, 0155 57543434, RYnlSPbEYh@bn.al, https://q8phlLSJgq.bn.al, N2AsWEMI5D 565, 859663 p0GdKDTbYV
  • Messages: 9FBqD2CNIJ (to|from) +49 155 75734343
  • Calls from and to +49 155 75734343 (Android only)
  • Calendar: fWAs4GFbpN, at urscf2178L, 2022-08-14T08:56 (iOS only)
  • Reminder: b5jHg3Eh1k, HQBOdx4kx2 (scheduled for 2022-08-02T13:38) (iOS only)
  • Health details: Name DkwIXobsJN t5TfTlezmn, DOB 1973-05-15, female, height 146cm, weight 108.5kg (iOS only)
  • Home data: Rooms bEZf1h06j1, DX7BgPtH99 (basement); second home g1bVNue3On (iOS only)
  • Device name: R2Gl5OLv20
  • Note: S0Ei7sFP9b (iOS only)
  • Phone number: +49 155 85834346 (Android only)
  • Location: Schreinerweg 6, 38126 Braunschweig; 52.235288, 10.564235
  • Apple ID (iOS only)
  • Clipboard: LDDsvPqQdT

LICENSE

This code is licensed under the MIT license, see the LICENSE file for details.

Status

This repository is archived in the state it was in when I submitted my thesis. I will not be updating it any further. However, I am working together with @zner0L on a follow-up project. As a part of this project, we will also make sure to extract reusable libraries and tools from this research code and document them more thouroughly. Follow me and datarequests.org on Mastodon to stay up to date on that.