Support Anbox Android Linker

Question

Support Anbox Android Linker

duraki opened this issue 4 years ago · 4 comments

As per our internal convo, @oleavr, we need frida-gum to assist us in spawning a frida-server for Anbox Android linker 🙌

Environment

So, anbox is like a proprietary/partly-open-sourced "gennymotion", but they still support emulation on more native layer through LXD containers. These containers run in cloud, based on Ubuntu Bionic (18.04) for aarch64. The android emulation (or anbox, really) is visible on "host" machine (host machine = Ubuntu aarch64). Therefore, processes from Android are visible in Ubuntu via e.g. ps aux.

So we can either:

Interact from Linux FS (by sshing to Ubuntu 18.04 aarch64) [tty: root@ams-c1gd8hhde9pihc932u0g]
Interact from Android FS (by exposing adb, or simply anbox-shell from withing Linux [tty: :/ #]

Linux FS

Ubuntu Bionic 18.04 aarch64
Host machine

Process system hierarchy (nesting) - host can see childs (android) processes:

root@ams-c1gd8hhde9pihc932u0g:/tmp# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.1  12856  5564 ?        Ss   Mar28   0:00 /sbin/init
root        41  0.0  0.2  17944  6904 ?        S<s  Mar28   0:00 /lib/systemd/systemd-journald
root        94  0.0  0.0   8324  2536 ?        Ss   Mar28   0:00 sudo -u anbox -g anbox --preserve-env=LIBC_FATAL_STDERR XDG_RUNTIME_DIR ANBOX_LOG_LEVEL ARG_REMOTE_ADDR ANBOX_ADB_LISTEN_ADDRESS NOTIFY_SOCKET
root       395  0.0  0.1  10420  3708 ?        Ss   Mar28   0:00 /usr/sbin/sshd -D
anbox      428  0.0  4.1 3920132 122524 ?      S<l  Mar28   5:27 /usr/bin/anbox session --platform=null --metrics-server=influxdb:192.168.100.1:8095,raw --log-level=info --extra-properties=/run/user/1000/anbo
...
100000     519  0.0  3.1 5386992 90944 ?       Sl   Mar28   0:00 zygote64
100000     520  0.0  2.6 1710392 78496 ?       Sl   Mar28   0:00 zygote
101000     522  0.0  0.1  38028  3580 ?        S<   Mar28   0:00 /system/bin/hw/android.hidl.allocator@1.0-service
101000     523  0.0  0.1  78800  4644 ?        S<l  Mar28   0:46 /system/bin/hw/android.system.suspend@1.0-service
100000     524  0.0  0.1  36420  3128 ?        S<   Mar28   0:00 /system/bin/healthd
101041     525  0.0  0.1  24288  4404 ?        S<l  Mar28   0:00 /vendor/bin/hw/android.hardware.audio@2.0-service
101041     526  0.0  0.1  76492  3508 ?        S<l  Mar28   0:00 /vendor/bin/hw/android.hardware.broadcastradio@1.1-service
...
101010     542  0.0  0.1  39580  4080 ?        S<   Mar28   0:00 /vendor/bin/hw/android.hardware.wifi@1.0-service

Android FS

Android Emulator
getprop ro.build.version.sdk => 29

modalias config

$ cat /sys/devices/system/cpu/modalias
cpu:type:aarch64:feature:,0000,0001,0002,0003,0004,0005,0006,0007,0008,0009,000A,000B,000C,000F,0010,0014,001C

Linker config from /proc/$PID/maps:

cat /proc/3489/maps  | grep linker
f7aa4000-f7ac1000 r--p 00000000 00:41 14457                              /var/snap/lxd/common/lxd/storage-pools/ams0/containers/ams-c1caemco83oipgb88mmg/rootfs/var/lib/anbox/rootfs/system/apex/com.android.runtime.release/bin/linker
f7ac1000-f7b58000 r-xp 0001d000 00:41 14457                              /var/snap/lxd/common/lxd/storage-pools/ams0/containers/ams-c1caemco83oipgb88mmg/rootfs/var/lib/anbox/rootfs/system/apex/com.android.runtime.release/bin/linker
f7b58000-f7b59000 rw-p 000b4000 00:41 14457                              /var/snap/lxd/common/lxd/storage-pools/ams0/containers/ams-c1caemco83oipgb88mmg/rootfs/var/lib/anbox/rootfs/system/apex/com.android.runtime.release/bin/linker
f7b59000-f7b5d000 r--p 000b5000 00:41 14457                              /var/snap/lxd/common/lxd/storage-pools/ams0/containers/ams-c1caemco83oipgb88mmg/rootfs/var/lib/anbox/rootfs/system/apex/com.android.runtime.release/bin/linker

Experiments

root@ams-c1gd8hhde9pihc932u0g:~# arch
aarch64

root@ams-c1gd8hhde9pihc932u0g:~# anbox-shell
:/ # getprop ro.product.cpu.abi
arm64-v8a

:/ # getprop
# => see below

full getprop gist

When running frida-server (aarch64) on Ubuntu aarch64

conclusion at the end...

root@ams-c1gd8hhde9pihc932u0g:/tmp# wget https://github.com/frida/frida/releases/download/14.0.6/frida-server-14.0.6-linux-arm64.xz -O frida-server.xz
...
Saving to: ‘frida-server.xz’

root@ams-c1gd8hhde9pihc932u0g:/tmp# unxz frida-server.xz
root@ams-c1gd8hhde9pihc932u0g:/tmp# chmod +x frida-server
root@ams-c1gd8hhde9pihc932u0g:/tmp# ./frida-server
./frida-server: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by ./frida-server)

Ugly fix:

root@ams-c1gd8hhde9pihc932u0g:/opt/glibc/build# cat /etc/ld.so.conf
include /etc/ld.so.conf.d/*.conf

root@ams-c1gd8hhde9pihc932u0g:/opt/glibc/build# cat /etc/ld.so.conf.d/*.conf
# Multiarch support
/usr/local/lib/aarch64-linux-gnu
/lib/aarch64-linux-gnu
/usr/lib/aarch64-linux-gnu
/usr/lib/aarch64-linux-gnu/libfakeroot
# libc default configuration
/usr/local/lib

root@ams-c1gd8hhde9pihc932u0g:/# cd /opt
root@ams-c1gd8hhde9pihc932u0g:/opt# export glibc_install="$(pwd)/glibc/build/install"
root@ams-c1gd8hhde9pihc932u0g:/opt# git clone git://sourceware.org/git/glibc.git && cd glibc
root@ams-c1gd8hhde9pihc932u0g:/opt/glibc# git checkout glibc-2.28
root@ams-c1gd8hhde9pihc932u0g:/opt/glibc# mkdir build && cd build
root@ams-c1gd8hhde9pihc932u0g:/opt/glibc/build# ../configure --prefix "$glibc_install"
root@ams-c1gd8hhde9pihc932u0g:/opt/glibc/build# make -j `nproc` # => then: make install -j `nproc`

root@ams-c1gd8hhde9pihc932u0g:/# ldd /tmp/frida-server
./frida-server-14.0.6-linux-arm64: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by ./frida-server-14.0.6-linux-arm64)
        linux-vdso.so.1 (0x0000ffffbb8a1000)
        libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000ffffbae4f000)
        libresolv.so.2 => /lib/aarch64-linux-gnu/libresolv.so.2 (0x0000ffffbae2a000)
        libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffffbad71000)
        libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffffbad45000)
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffffbabec000)
        /lib/ld-linux-aarch64.so.1 (0x0000ffffbb875000)

root@ams-c1gd8hhde9pihc932u0g:/opt/glibc/build# patchelf --set-interpreter /opt/glibc/build/install/lib/ld-linux-aarch64.so.1 --set-rpath /opt/glibc/build/libc.so.6 /tmp/frida-server
warning: working around a Linux kernel bug by creating a hole of 73728 bytes in ‘/tmp/frida-server’

root@ams-c1gd8hhde9pihc932u0g:/tmp# ./frida-server -l 0.0.0.0 -v
# => stdout empty

# => I tried connecting from my PC but nop.
frida-ps --host=3.120.235.48 -a
[... after some time ...]: connect to 3.120.235.48: Operation timed out

root@ams-c1gd8hhde9pihc932u0g:/# echo "add-auto-load-safe-path /opt/glibc/build/install/lib/libthread_db-1.0.so" >> ~/.gdbinit

root@ams-c1gd8hhde9pihc932u0g:/tmp# gdb ./frida-server
...
(gdb) run
Starting program: /tmp/frida-server
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/opt/glibc/build/install/lib/libthread_db.so.1".
[New Thread 0xfffff7d441e0 (LWP 4911)]

^C // forced

Thread 1 "frida-server" received signal SIGINT, Interrupt.
0x0000fffff7e28ac0 in __GI___poll (fds=0xaaaaab4e0150, nfds=2, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
41	  return SYSCALL_CANCEL (ppoll, fds, nfds, timeout_ts_p, NULL, 0);
(gdb) bt
#0  0x0000fffff7e28ac0 in __GI___poll (fds=0xaaaaab4e0150, nfds=2, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
#1  0x0000aaaaab2a3698 in ?? ()
#2  0x0000aaaaab2a39cc in ?? ()
#3  0x0000aaaaaaacdea0 in ?? ()
#4  0x0000aaaaaaace5f8 in ?? ()
#5  0x0000aaaaaaaceabc in ?? ()
#6  0x0000fffff7d85aa0 in __libc_start_main (main=0x0, argc=0, argv=0x0, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:308
#7  0x0000aaaaaaaca244 in ?? ()
Backtrace stopped: not enough registers or memory available to unwind further

Some more experiments with frida-tools via pip3:

root@ams-c1gd8hhde9pihc932u0g:/tmp# pip3 install frida-tools
Successfully built frida-tools frida

# => trying to attach to *any* when frida-server is running on host
root@ams-c1gd8hhde9pihc932u0g:/tmp# frida-trace -H 0.0.0.0:9991 -i "recv*" com.google.android.gms

***
Failed to load the Frida native extension: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /usr/local/lib/python3.6/dist-packages/_frida.cpython-36m-aarch64-linux-gnu.so)
Please ensure that the extension was compiled for Python 3.x.
***

Traceback (most recent call last):
  File "/usr/local/bin/frida-trace", line 7, in <module>
    from frida_tools.tracer import main
  File "/usr/local/lib/python3.6/dist-packages/frida_tools/tracer.py", line 13, in <module>
    import frida
  File "/usr/local/lib/python3.6/dist-packages/frida/__init__.py", line 24, in <module>
    raise ex
  File "/usr/local/lib/python3.6/dist-packages/frida/__init__.py", line 7, in <module>
    import _frida
ImportError: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /usr/local/lib/python3.6/dist-packages/_frida.cpython-36m-aarch64-linux-gnu.so)

# => what if we ldd few more things?
root@ams-c1gd8hhde9pihc932u0g:/tmp# ldd /usr/bin/python3
        linux-vdso.so.1 (0x0000ffff9d9d9000)
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff9d854000)
        libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffff9d828000)
        libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000ffff9d813000)
        libutil.so.1 => /lib/aarch64-linux-gnu/libutil.so.1 (0x0000ffff9d800000)
        libexpat.so.1 => /lib/aarch64-linux-gnu/libexpat.so.1 (0x0000ffff9d7c1000)
        libz.so.1 => /lib/aarch64-linux-gnu/libz.so.1 (0x0000ffff9d794000)
        libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffff9d6db000)
        /lib/ld-linux-aarch64.so.1 (0x0000ffff9d9ad000)

root@ams-c1gd8hhde9pihc932u0g:/tmp# ldd /usr/local/lib/python3.6/dist-packages/_frida.cpython-36m-aarch64-linux-gnu.so
/usr/local/lib/python3.6/dist-packages/_frida.cpython-36m-aarch64-linux-gnu.so: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /usr/local/lib/python3.6/dist-packages/_frida.cpython-36m-aarch64-linux-gnu.so)
        linux-vdso.so.1 (0x0000ffff882da000)
        libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000ffff86825000)
        libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000ffff867f9000)
        libresolv.so.2 => /lib/aarch64-linux-gnu/libresolv.so.2 (0x0000ffff867d4000)
        libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000ffff8671b000)
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000ffff865c2000)
        /lib/ld-linux-aarch64.so.1 (0x0000ffff882ae000)

Conclusion: firda-server is indeed somehow spawned on Ubuntu 18.04 aarch64 by fooling the lib checks. For some reason tho, it does not work as expected. Frida server should be running, and it should accept remote connections. Instead, app just stays in process environment and waits...

When running frida-server (arm64-v8a) on Android shell

conclusion at the end...

Next I tried running arm64-v8a version of frida-server from with-in Android shell:

$ root@ams-c1gd8hhde9pihc932u0g:/tmp# wget https://github.com/frida/frida/releases/download/14.2.14/frida-server-14.2.14-android-arm64.xz -O frida-server.xz
$ unxz frida-server.xz
$ adb push frida-server /data/local/tmp 

$ root@ams-c1gd8hhde9pihc932u0g:/# anbox-shell
# => now in Android emulator
:/ # cd /data/local/tmp

:/data/local/tmp # ./frida-server
Aborted (core dumped)
# => exits, @see adb logcat below
# https://gist.githubusercontent.com/duraki/a39b9bef582537eecbc1dcded45b8144/raw/12d03772112d8c25c89867a35a66598a0681d710/logcat.txt

:/data/local/tmp # ../frida-server --version
14.0.6
# => --version works tho

Conclusion: firda-server does not work inside Androids emulator. Although, printing frida-server --version works fine, trying to bring host/server up does not work due to Androids Linker bug (as indicated by adb logcat).

Resources:

BTW. I may be able to provide ssh pivot to the container(s).

Answer 1 · 2021-04-01T09:44:49.000Z

Sounds like your issue is that the binaries are built for Ubuntu 20.04, but you’re running on Ubuntu 18.04. OS suggest you try following the instructions for compiling from source.

https://frida.re/docs/building/

Frida-server is built as part of Frida core. The example uses mips, but I suspect you’ll want to use armhf. Make sure you build the SDKs from source too, the build system may try to download the prebuilt versions if you don’t manually build it before building Frida itself.

Answer 2 · 2021-04-01T15:41:07.000Z

/me already aware of that. See frida/frida-ci/issues/2

Answer 3 · 2021-04-01T17:44:54.000Z

Anbox Cloud runs LXD containers in an LXD cluster. In our case the cluster is comprised strictly of arm64 hosts running arm64 containers. Each of these containers runs Ubuntu 18:04 that provides a runtime environment around Anbox that you can customize for streaming, automation, instrumentation, etc. Anbox in turn abstracts hardware access and creates an LXD container nested inside where Android (standard AOSP Android 10) is actually running.

Sounds like your issue is that the binaries are built for Ubuntu 20.04, but you’re running on Ubuntu 18.04. OS suggest you try following the instructions for compiling from source.

Yes, that is the first problem at the surface. Myself I tried a 20.04 chroot (within the 18.04 container) from which to install frida and frida-tools:

curl -fsSL https://cloud-images.ubuntu.com/releases/focal/release/ubuntu-20.04-server-cloudimg-arm64-root.tar.xz -o focal.tar.xz
unxz focal.tar.xz

mkdir -p /mnt/focal
tar -xf focal.tar --exclude=dev -C /mnt/focal

mkdir -p /mnt/focal/proc && mount -t proc /proc /mnt/focal/proc
mkdir -p /mnt/focal/sys && mount -t sysfs /sys /mnt/focal/sys
mkdir -p /mnt/focal/dev && mount -o bind /dev /mnt/focal/dev
mkdir -p /mnt/focal/dev/pts && mount -o bind /dev/pts /mnt/focal/dev/pts
rm /mnt/focal/etc/resolv.conf && cat /etc/resolv.conf > /mnt/focal/etc/resolv.conf

chroot /mnt/focal /bin/bash -c "apt update && apt install -y python3-pip"
chroot /mnt/focal /bin/bash -c "pip3 install frida frida-tools"
chroot /mnt/focal /bin/bash -c "frida-ps"

At this point all the Frida stuff is running in a 20.04 chroot with the expected library dependencies. I can at least invoke the frida CLI tools and look around a bit. However, when tracing or attaching to an Android process (still within the 20.04 chroot) I get problems

Failed to attach: unable to inject library into process without libc

I guess chaining libc dependencies like that (tree view below) doesn't make sense

- Ubuntu 18.04 LXD container
  - Android LXD container
  - Ubuntu 20.04 chroot
    - Frida
    - Frida tools

suggest you try following the instructions for compiling from source.
https://frida.re/docs/building/

We also tried that for a while but also had difficulty along the way, at which point we thought to outsource help for this task.

Answer 4 · 2021-04-01T18:42:13.000Z

In order for the 20.04 chroot to work, you will need to make sure it bind mounts the /proc file system.

Also the Frida tools and target will need to both be running in the chroot (well with the same file and path for libc). Frida will use /proc/pid/maps on itself and the target to find the base address of libc. Then calculate the relative virtual address of various functions in its own process and add it to the base of libc in the target process to find where the same functions reside in memory in the target.

This will obviously need libc in each to be binary identical. Hope this helps.