gojue/ecapture

PCAP mode can only decrypt partial HTTPS traffic, not all of it

zaxtyson opened this issue · 8 comments

zaxtyson commented

Describe the bug
text 模式下可以正常抓取到 APP 的 HTTPS 请求以及响应数据（gzip），由于 gzip 数据不便观察，我切换到 pcap 模式抓取，以便使用 Wirshark 进行分析。然而抓取的 pcapng 文件中，大部分数据是未解密的 TLS 数据包而非解密后的 HTTPS 数据，只有少量解密的 HTTPS 数据。

Screenshots
1.抓取某个 APP 的数据，这里以酷安为例。

xagapro:/data/local/tmp # ps -A | grep coolapk
u0_a240      12860   988 28623504 524512 do_epoll_wait      0 S com.coolapk.market

2.使用 text 模式抓包，可以看到解密后的 HTTPS 数据。从 response headers 看来，确实是酷安服务器的 response。

xagapro:/data/local/tmp # ./ecapture_0.7.2 tls --pid=12860
2024/01/24 14:52:02 Your environment is like a container. We won't be able to detect the BTF configuration.
tls_2024/01/24 14:52:02 ECAPTURE :: ecapture Version : androidgki_aarch64:v0.7.2:5.4.0-155-generic
tls_2024/01/24 14:52:02 ECAPTURE :: Pid Info : 15711
tls_2024/01/24 14:52:02 ECAPTURE :: Kernel Info : 5.10.101
tls_2024/01/24 14:52:02 EBPFProbeOPENSSL        module initialization
tls_2024/01/24 14:52:02 EBPFProbeOPENSSL        master key keylogger:
tls_2024/01/24 14:52:02 ECAPTURE ::     Module.Run()
tls_2024/01/24 14:52:02 EBPFProbeOPENSSL        Text MODEL
tls_2024/01/24 14:52:02 EBPFProbeOPENSSL        OpenSSL/BoringSSL version found, ro.build.version.release=13
tls_2024/01/24 14:52:02 EBPFProbeOPENSSL        HOOK type:2, binrayPath:/apex/com.android.conscrypt/lib64/libssl.so
tls_2024/01/24 14:52:02 EBPFProbeOPENSSL        Hook masterKey function:SSL_in_init
tls_2024/01/24 14:52:02 EBPFProbeOPENSSL        libPthread:/apex/com.android.runtime/lib64/bionic/libc.so
tls_2024/01/24 14:52:02 EBPFProbeOPENSSL        target PID:12860
tls_2024/01/24 14:52:02 EBPFProbeOPENSSL        target all users.
tls_2024/01/24 14:52:02 EBPFProbeOPENSSL        BPF bytecode filename:user/bytecode/boringssl_a_13_kern.o
tls_2024/01/24 14:52:02 EBPFProbeOPENSSL        perfEventReader created. mapSize:4 MB
tls_2024/01/24 14:52:02 EBPFProbeOPENSSL        perfEventReader created. mapSize:4 MB
tls_2024/01/24 14:52:02 EBPFProbeOPENSSL        module started successfully.
tls_2024/01/24 14:52:02 ECAPTURE ::     start 1 modules
...
tls_2024/01/24 14:52:57         notice: SSLDataEvent's fd is 0.  pid:12860, fd:0, addr:
tls_2024/01/24 14:52:57 PID:12860, Comm:RxIoScheduler-1, TID:15698, Version:TLS_VERSION_UNKNOW_1073742595, Recived 508 bytes from 0.0.0.0, Payload:
HTTP/1.1 200 OK
Date: Wed, 24 Jan 2024 14:52:58 GMT
Content-Type: application/json; charset=UTF-8
Vary: Accept-Encoding
X-Powered-By: PHP/7.4.33
Set-Cookie: SESSID=xxxx; path=/; domain=.coolapk.com
X-Frame-Options: Deny
Content-Encoding: gzip
Server: Lego Server
X-Cache-Lookup: Cache Miss
Cache-Control: private, must-revalidate, max-age=0
Transfer-Encoding: chunked
X-NWS-LOG-UUID: 11839391835786498221
Connection: keep-alive
X-Cache-Lookup: Cache Miss


tls_2024/01/24 14:52:57         notice: SSLDataEvent's fd is 0.  pid:12860, fd:0, addr:
tls_2024/01/24 14:52:57 PID:12860, Comm:RxIoScheduler-1, TID:15698, Version:TLS_VERSION_UNKNOW_1073742595, Recived 5 bytes from 0.0.0.0, Payload:
929

tls_2024/01/24 14:52:57         notice: SSLDataEvent's fd is 0.  pid:12860, fd:0, addr:
tls_2024/01/24 14:52:57 PID:12860, Comm:RxIoScheduler-1, TID:15698, Version:TLS_VERSION_UNKNOW_1073742595, Recived 2345 bytes from 0.0.0.0, Payload:
�Zms�F�+��ȗ��⸮I�W
!8���:éS��4��J�Ü�v(���3#�vmö����b-���t?���ORъ���I�N����s>��Wk99�������/D���d2�4UW'2�_;zp�϶��kdÅ�⸮��;WZ����΋�
Ihe�
�-�Y�É�=&�Yz�T=a�h�FA����)��YtZ���hY�R�ۂF�Ö����Y�Jf��VW��:OP�l���ö6��+��GIUbå����lÅWi��QQ-��̶lg>��c�g��)��E�yaX?�u��Go�
j��
...

3.切换到 pcap 模式抓包

xagapro:/data/local/tmp # ./ecapture_0.7.2 tls -m pcap --pcapfile=coolapk.pcapng --pid=12860
2024/01/24 15:01:42 Your environment is like a container. We won't be able to detect the BTF configuration.
tls_2024/01/24 15:01:42 ECAPTURE :: ecapture Version : androidgki_aarch64:v0.7.2:5.4.0-155-generic
tls_2024/01/24 15:01:42 ECAPTURE :: Pid Info : 16479
tls_2024/01/24 15:01:42 ECAPTURE :: Kernel Info : 5.10.101
tls_2024/01/24 15:01:42 EBPFProbeOPENSSL        module initialization
tls_2024/01/24 15:01:42 ECAPTURE ::     Module.Run()
tls_2024/01/24 15:01:42 EBPFProbeOPENSSL        Pcapng MODEL
tls_2024/01/24 15:01:42 EBPFProbeOPENSSL        OpenSSL/BoringSSL version found, ro.build.version.release=13
tls_2024/01/24 15:01:42 EBPFProbeOPENSSL        HOOK type:2, binrayPath:/apex/com.android.conscrypt/lib64/libssl.so
tls_2024/01/24 15:01:42 EBPFProbeOPENSSL        Ifname:wlan0, Ifindex:38,  Port:443, Pcapng filepath:/data/local/tmp/coolapk.pcapng
tls_2024/01/24 15:01:42 EBPFProbeOPENSSL        Hook masterKey function:SSL_in_init
tls_2024/01/24 15:01:42 EBPFProbeOPENSSL        target PID:12860
tls_2024/01/24 15:01:42 EBPFProbeOPENSSL        target all users.
tls_2024/01/24 15:01:42 EBPFProbeOPENSSL        BPF bytecode filename:user/bytecode/boringssl_a_13_kern.o
tls_2024/01/24 15:01:42 EBPFProbeOPENSSL        saving pcapng file: /data/local/tmp/coolapk.pcapng
tls_2024/01/24 15:01:42 EBPFProbeOPENSSL        perfEventReader created. mapSize:4 MB
tls_2024/01/24 15:01:42 EBPFProbeOPENSSL        perfEventReader created. mapSize:4 MB
tls_2024/01/24 15:01:42 EBPFProbeOPENSSL        module started successfully.
tls_2024/01/24 15:01:42 ECAPTURE ::     start 1 modules
tls_2024/01/24 15:01:46 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM 7b1668f91e7aa197087ac50e3156517d4bba5408ff17a7f9cd00b224067e4f48 to file success, 938 bytes
tls_2024/01/24 15:01:46 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM 2ea92a5da6b6fadaf45861529380a1c57988b3a57e06021cd4edcb0c19335a0f to file success, 938 bytes
tls_2024/01/24 15:01:46 EBPFProbeOPENSSL        save pcapng success, count:100
tls_2024/01/24 15:01:48 EBPFProbeOPENSSL        save pcapng success, count:88
tls_2024/01/24 15:01:49 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM 90048d3e9b6c7998f37d8bf3a137d04ccad132bcc3eefeb4cce7f17c80278bd3 to file success, 938 bytes
tls_2024/01/24 15:01:49 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM e72dacdba006bc1b703f14a0730a8c23c291ddc834d1c18d101c83f600f342e9 to file success, 938 bytes
tls_2024/01/24 15:01:49 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM ca282201ab7a1da13fd97ac7a660249c31f4a8d0d92ac3aeb2acbbc87cd88fc3 to file success, 938 bytes
tls_2024/01/24 15:01:50 EBPFProbeOPENSSL        save pcapng success, count:733
tls_2024/01/24 15:01:52 EBPFProbeOPENSSL        save pcapng success, count:416
tls_2024/01/24 15:01:54 EBPFProbeOPENSSL        save pcapng success, count:241
tls_2024/01/24 15:01:56 EBPFProbeOPENSSL        save pcapng success, count:253
tls_2024/01/24 15:01:57 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM b5c244fcc8512c609a272944572ab1d07e91f85bba845c295eb155de39ce1c44 to file success, 938 bytes
tls_2024/01/24 15:01:58 EBPFProbeOPENSSL        save pcapng success, count:230
tls_2024/01/24 15:02:00 EBPFProbeOPENSSL        save pcapng success, count:22
tls_2024/01/24 15:02:00 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM 9d38163a4b1ad47cc06a7d49fef1a0346722147fc923cfb4cc423171ec2bb2c6 to file success, 938 bytes
tls_2024/01/24 15:02:02 EBPFProbeOPENSSL        save pcapng success, count:7934
tls_2024/01/24 15:02:04 EBPFProbeOPENSSL        save pcapng success, count:8
tls_2024/01/24 15:02:06 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM 1969d672b105b41ebe4df519c7c8608a5668e76253ad8f67fb6a0c429671e69a to file success, 938 bytes
tls_2024/01/24 15:02:06 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM 3816443af5ded88c579a6531d9b69838eb1998eecfc9296d9e0efd2f2eb7d302 to file success, 938 bytes
tls_2024/01/24 15:02:06 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM 878581ffe9f20e281a3ca7e6e2df52242002edd2efd87c36caedcdb81ba25ec0 to file success, 938 bytes
tls_2024/01/24 15:02:06 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM 1436dcb83d798cd0812abfe01ae7d82ff280a9ed362a1dd757e570cdda84e2d1 to file success, 938 bytes
tls_2024/01/24 15:02:06 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM 1e36f711dbdad7d3a3c154dc0e6cbbe6f2a579b9882847d50584d4a5ee951add to file success, 938 bytes
tls_2024/01/24 15:02:06 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM f4525f2661c8f2bd59300f629313cb5745e407866687a61d408b6f7908b54d59 to file success, 938 bytes
tls_2024/01/24 15:02:06 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM 35b33bc6fb1165f6d219cffcd8cef695ed83b3185b2b3916ddf5868dc258fe11 to file success, 938 bytes
tls_2024/01/24 15:02:06 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM 7ba3a45604bc020d594dd64139fe9bd70ef593359e4973a3c3dadc68f946b4e1 to file success, 938 bytes
tls_2024/01/24 15:02:07 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM ea1b0c4f7b4222c947e6dab8c9a49fafdb1ccdbd5ea2e9b7b43a8d28173a04a3 to file success, 938 bytes
tls_2024/01/24 15:02:08 EBPFProbeOPENSSL        save pcapng success, count:3207
tls_2024/01/24 15:02:09 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM 36d156fb9dc520c211994a8b6dde318eb35645af4d3483cc515344f9de4eb245 to file success, 938 bytes
tls_2024/01/24 15:02:10 EBPFProbeOPENSSL        save pcapng success, count:266
tls_2024/01/24 15:02:10 EBPFProbeOPENSSL        TLS1_3_VERSION: save CLIENT_RANDOM 60313f2b9a1adffc6d46ede3277327d665ce53c4285be6557fa93e3149448f06 to file success, 938 bytes
tls_2024/01/24 15:02:12 EBPFProbeOPENSSL        save pcapng success, count:25
ÜCtls_2024/01/24 15:02:13 EBPFProbeOPENSSL      close.
tls_2024/01/24 15:02:13 EBPFProbeOPENSSL        close

xagapro:/data/local/tmp # ls -hal
total 37M
drwxrwx--x 2 shell shell 3.3K 2024-01-24 23:06 .
drwxr-x--x 7 root  root  3.3K 2023-06-10 21:27 ..
-rw-r--r-- 1 root  root   13M 2024-01-24 23:02 coolapk.pcapng
-rwxrwxrwx 1 root  root   12M 2023-11-19 23:45 ecapture_0.6.6
-rwxrwxrwx 1 root  root   12M 2024-01-02 00:05 ecapture_0.7.2

可以抓到部分 HTTPS 数据，但是观察发现这些请求都来自 APP 内的广告 SDK，而非 APP 本身的请求。

image

大部分请求仍然是未解密的 TLS 流量，这点让我很疑惑，既然 text模式下已经可以看到解密后的数据，为何 pcapng 中会是加密状态。

image

Linux Server/Android (please complete the following information):

xagapro:/ # zcat /proc/config.gz | grep CONFIG_DEBUG_INFO_BTF
# CONFIG_DEBUG_INFO_BTF is not set
xagapro:/ # uname -a
Linux localhost 5.10.101-android12-9-00027-g1292f517889e-ab8602202 #1 SMP PREEMPT Mon May 16 11:18:04 UTC 2022 aarch64 Toybox

设备为 Redmi Note11T Pro+，Android12，内核 5.10.101，内核没有打开 BTF，因此使用了 CO-RE 版本的 ecapture v0.7.2 抓包

Additional context
尝试过降级到 v0.6.6，结果与 v0.7.2 相同

cfc4n commented

大概是APP没用使用系统默认的ssl链接库/apex/com.android.conscrypt/lib64/libssl.so。

检查一下app的ssl类库是哪个，之后通过 --libssl参数指定。

zaxtyson commented

如果说是 libssl 库不对，那么 text 模式同样是使用 /apex/com.android.conscrypt/lib64/libssl.so，为什么能正确抓到 APP 的数据呢

cfc4n commented

嗯，有说不通的地方。 有没有可能，他两个类库都用了。只是个别接口走的系统默认的，大部分业务逻辑接口用的不是系统默认类库？

还是建议你先分析apk使用的ssl类库。

zaxtyson commented

APP 自带了一个 libttboringssl.so 库，我尝试指定该库为 libssl 的路径，这次不仅 pcap 没有解密数据了，text 模式也不输出了。使用系统默认的库，text 模式是十分正常的，能看到 APP 发出的请求及响应。

其实我疑惑的点在于，理论上 text 模式和 pcap 模式的行为应当是一致的，pcap 模式做的只是将 text 模式提取到的数据以 pcapng 的格式存储下来。既然 text 模式正常解密了所有的流量，没有理由 pcap dump下来的数据有一部分是未解密状态。🤷‍♂️

cfc4n commented

text模式跟pcap模式应该是一样的，而且从日志里也看到打印了CLIENT_RANDOM 事件，应该都捕获正确了。 还有可能就是wireshark的解析这里除了问题，尝试升级一下版本试试。 我常用的版本是4.2.2

zaxtyson commented

嗯嗯，我也是用的 4.2.2 版本 Wireshark.

TLS1_3_VERSION: save CLIENT_RANDOM ca282201ab7a1da13fd97ac7a660249c31f4a8d0d92ac3aeb2acbbc87cd88fc3 to file success, 938 bytes
tls_2024/01/24 15:01:50 EBPFProbeOPENSSL        save pcapng success, count:733
tls_2024/01/24 15:01:52 EBPFProbeOPENSSL        save pcapng success, count:416

pcap 模式的日志中，除了 save CLIENT_RANDOM，更多的是 save pcapng success。出现 CLIENT_RANDOM的应该是解密了（对应Wireshark中看到的少量 HTTPS 包），而只显示 save pcapng success 似乎是没有没有解密成功。

cfc4n commented

是的，密钥应该都拿到了，你也可以通过查看目录下ecapture_masterkey.log文件来确认密钥是否正确。

你也可以使用keylog模式，来保存SSLKEY，再用tshark实时解密，对比跟wireshark的结果。

ref: https://github.com/gojue/ecapture?tab=readme-ov-file#keylog-mode

cfc4n commented

ping ?