在多种操作系统发行版上无法捕获mysqld服务端的keylog
Closed this issue · 6 comments
BoltzmannWXD commented
在多种操作系统发行版上无法捕获mysqld服务端的keylog。
目前尝试过的操作系统有CentOS8、OpenEuler22、Ubuntu22。
具体信息如下：
CentOS8
Operating System: CentOS Linux 8
CPE OS Name: cpe:/o:centos:centos:8
Kernel: Linux 4.18.0-553.5.1.el8.x86_64
Architecture: x86-64
OpenEuler22
Operating System: openEuler 22.03 (LTS-SP3)
Kernel: Linux 5.10.0-182.0.0.95.oe2203sp3.x86_64
Architecture: x86-64
Ubuntu22
Operating System: Ubuntu 22.04.4 LTS
Kernel: Linux 5.15.0-117-generic
Architecture: x86-64
问题复现
在服务端开启ecapture
./ecapture tls -m key
然后再通过其他服务器的命令进行连接
mysql -h 192.168.xx.xx -u admin -p
发现无法正常捕获keylog。
查看debug日志效果如下：
sudo cat /sys/kernel/debug/tracing/trace_pipe
connection-2342 [000] ....1 1735.963771: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :2295
connection-2342 [000] ....1 1735.963783: bpf_trace_printk: TLS version :772
connection-2342 [000] ....1 1735.963783: bpf_trace_printk: client_random: 0 0 0
connection-2342 [000] ....1 1735.963784: bpf_trace_printk: cipher_suite_st pointer: 2f8
connection-2342 [000] ....1 1735.963787: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0
connection-2342 [000] ....1 1735.963788: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14
connection-2342 [000] ....1 1735.963820: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :2295
connection-2342 [000] ....1 1735.963821: bpf_trace_printk: TLS version :772
connection-2342 [000] ....1 1735.963821: bpf_trace_printk: client_random: 0 0 0
connection-2342 [000] ....1 1735.963822: bpf_trace_printk: cipher_suite_st pointer: 2f8
connection-2342 [000] ....1 1735.963823: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0
connection-2342 [000] ....1 1735.963824: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14
connection-2342 [000] ....1 1735.963831: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :2295
connection-2342 [000] ....1 1735.963831: bpf_trace_printk: TLS version :772
connection-2342 [000] ....1 1735.963832: bpf_trace_printk: client_random: 0 0 0
connection-2342 [000] ....1 1735.963832: bpf_trace_printk: cipher_suite_st pointer: 2f8
connection-2342 [000] ....1 1735.963833: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0
connection-2342 [000] ....1 1735.963834: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14
connection-2342 [000] ....1 1735.963840: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :2295
connection-2342 [000] ....1 1735.963840: bpf_trace_printk: TLS version :772
connection-2342 [000] ....1 1735.963841: bpf_trace_printk: client_random: 0 0 0
connection-2342 [000] ....1 1735.963841: bpf_trace_printk: cipher_suite_st pointer: 2f8
connection-2342 [000] ....1 1735.963842: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0
connection-2342 [000] ....1 1735.963843: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14
connection-2342 [000] ....1 1735.963874: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :2295
connection-2342 [000] ....1 1735.963875: bpf_trace_printk: TLS version :772
connection-2342 [000] ....1 1735.963875: bpf_trace_printk: client_random: 0 0 0
connection-2342 [000] ....1 1735.963875: bpf_trace_printk: cipher_suite_st pointer: 2f8
connection-2342 [000] ....1 1735.963876: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0
connection-2342 [000] ....1 1735.963877: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14
实际发起的tls连接版本是tls1.2，而捕获的却是772（tls.13）
正常情况
通过当前机器发起的mysql连接是可以正常获取keylog的，查看debug日志效果如下：
mysql-4860 [000] d... 111590.766726: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :4860
mysql-4860 [000] d... 111590.766745: bpf_trace_printk: TLS version :772
mysql-4860 [000] d... 111590.766746: bpf_trace_printk: client_random: 42 51 38
mysql-4860 [000] d... 111590.766747: bpf_trace_printk: cipher_suite_st pointer: 2baf6d8
mysql-4860 [000] d... 111590.766747: bpf_trace_printk: cipher_suite_st value: 34137b70
mysql-4860 [000] d... 111590.766748: bpf_trace_printk: cipher_id: 50336514
mysql-4860 [000] d... 111590.766749: bpf_trace_printk: *****master_secret*****: 0 0 0
mysql-4860 [000] d... 111590.766818: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :4860
mysql-4860 [000] d... 111590.766819: bpf_trace_printk: TLS version :772
mysql-4860 [000] d... 111590.766820: bpf_trace_printk: client_random: 42 51 38
mysql-4860 [000] d... 111590.766821: bpf_trace_printk: cipher_suite_st pointer: 2baf6d8
mysql-4860 [000] d... 111590.766821: bpf_trace_printk: cipher_suite_st value: 34137b70
mysql-4860 [000] d... 111590.766821: bpf_trace_printk: cipher_id: 50336514
mysql-4860 [000] d... 111590.766822: bpf_trace_printk: *****master_secret*****: 0 0 0
mysql-4860 [000] d... 111590.766848: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :4860
mysql-4860 [000] d... 111590.766848: bpf_trace_printk: TLS version :772
mysql-4860 [000] d... 111590.766849: bpf_trace_printk: client_random: 42 51 38
mysql-4860 [000] d... 111590.766849: bpf_trace_printk: cipher_suite_st pointer: 2bae998
mysql-4860 [000] d... 111590.766849: bpf_trace_printk: cipher_suite_st value: 34137b70
mysql-4860 [000] d... 111590.766850: bpf_trace_printk: cipher_id: 50336514
mysql-4860 [000] d... 111590.766850: bpf_trace_printk: *****master_secret*****: 0 0 0
mysql-4860 [000] d... 111590.766857: bpf_trace_printk: openssl uprobe/SSL_write masterKey PID :4860
mysql-4860 [000] d... 111590.766857: bpf_trace_printk: TLS version :772
mysql-4860 [000] d... 111590.766857: bpf_trace_printk: client_random: 42 51 38
mysql-4860 [000] d... 111590.766858: bpf_trace_printk: cipher_suite_st pointer: 2bae998
mysql-4860 [000] d... 111590.766858: bpf_trace_printk: cipher_suite_st value: 34137b70
mysql-4860 [000] d... 111590.766858: bpf_trace_printk: cipher_id: 50336514
mysql-4860 [000] d... 111590.766859: bpf_trace_printk: *****master_secret*****: 0 0 0
该情况下发起的是tls1.3捕获的也是tls1.3，完全正确。
后续又在其他机器（CentOS8）部署了mysql-client和ecapture，发现能够正常捕获mysql-client发起的请求连接的keylog。
编译调试模式
刚开始尝试的时候用的是仓库里发行的 ecapture-v0.8.4-linux-amd64.tar.gz，后来发现无法正常使用，故在Ubuntu22上用代码进行了编译，并打开了debug模式：
make debug=1
发现了该问题。
结论
由此可以发现作为mysql客户端是可以正常捕获keylog的，但是作为服务端无法正常捕获mysqld的keylog。
cfc4n commented
请给一些详细的技术分析，讲清楚为什么mysqld的keylog模式无法生效，谢谢。
Please provide a detailed technical analysis, explaining clearly why the keylog mode of mysqld cannot take effect, thank you.
BoltzmannWXD commented
具体来说不仅限于mysqld，而是作为openssl服务端的问题。在gdb跟踪调试了openssl的SSL_accept函数后发现，只有在SSL_do_handshake函数返回后SSL参数才有正确的赋值。随后将openssl_masterkey.h里的"uprobe/SSL_write_key"调整成"uretprobe/SSL_write_key"，验证成功。
BoltzmannWXD commented
以下是调试服务端时候的日志
connection-38592 [003] d... 410734.666741: bpf_trace_printk: uprobe/SSL_write_key
connection-38592 [003] d... 410734.666761: bpf_trace_printk: new key :96c0, new: 14040
connection-38592 [003] d... 410734.666774: bpf_trace_printk: uretprobe/SSL_write_key
connection-38592 [003] d... 410734.666774: bpf_trace_printk: openssl uretprobe/SSL_write masterKey PID :38521
connection-38592 [003] d... 410734.666781: bpf_trace_printk: TLS version :772
connection-38592 [003] d... 410734.666782: bpf_trace_printk: client_random: 0 0 0
connection-38592 [003] d... 410734.666782: bpf_trace_printk: cipher_suite_st pointer: 1f0
connection-38592 [003] d... 410734.666786: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0
connection-38592 [003] d... 410734.666787: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14
connection-38592 [003] d... 410734.666790: bpf_trace_printk: uprobe/SSL_write_key
connection-38592 [003] d... 410734.666794: bpf_trace_printk: uretprobe/SSL_write_key
connection-38592 [003] d... 410734.666794: bpf_trace_printk: openssl uretprobe/SSL_write masterKey PID :38521
connection-38592 [003] d... 410734.666795: bpf_trace_printk: TLS version :772
connection-38592 [003] d... 410734.666795: bpf_trace_printk: client_random: 0 0 0
connection-38592 [003] d... 410734.666796: bpf_trace_printk: cipher_suite_st pointer: 1f0
connection-38592 [003] d... 410734.666797: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0
connection-38592 [003] d... 410734.666798: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14
connection-38592 [003] d... 410734.666815: bpf_trace_printk: uprobe/SSL_write_key
connection-38592 [003] d... 410734.666820: bpf_trace_printk: uprobe/SSL_write_key
connection-38592 [003] d... 410734.666823: bpf_trace_printk: uretprobe/SSL_write_key
connection-38592 [003] d... 410734.666824: bpf_trace_printk: openssl uretprobe/SSL_write masterKey PID :38521
connection-38592 [003] d... 410734.666824: bpf_trace_printk: TLS version :772
connection-38592 [003] d... 410734.666824: bpf_trace_printk: client_random: 0 0 0
connection-38592 [003] d... 410734.666825: bpf_trace_printk: cipher_suite_st pointer: 1f0
connection-38592 [003] d... 410734.666826: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0
connection-38592 [003] d... 410734.666826: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14
connection-38592 [003] d... 410734.666837: bpf_trace_printk: uprobe/SSL_write_key
connection-38592 [003] d... 410734.666840: bpf_trace_printk: uretprobe/SSL_write_key
connection-38592 [003] d... 410734.666840: bpf_trace_printk: openssl uretprobe/SSL_write masterKey PID :38521
connection-38592 [003] d... 410734.666841: bpf_trace_printk: TLS version :772
connection-38592 [003] d... 410734.666841: bpf_trace_printk: client_random: 0 0 0
connection-38592 [003] d... 410734.666842: bpf_trace_printk: cipher_suite_st pointer: 1f0
connection-38592 [003] d... 410734.666843: bpf_trace_printk: bpf_probe_read ssl_cipher_st_ptr failed, ret :-14, address:0
connection-38592 [003] d... 410734.666844: bpf_trace_printk: bpf_probe_read SSL_SESSION_ST_CIPHER_ID failed from SSL_SESSION->cipher_id, ret :-14
connection-38592 [003] d... 410734.672146: bpf_trace_printk: uretprobe/SSL_write_key
connection-38592 [003] d... 410734.672164: bpf_trace_printk: openssl uretprobe/SSL_write masterKey PID :38521
connection-38592 [003] d... 410734.672168: bpf_trace_printk: TLS version :771
connection-38592 [003] d... 410734.672169: bpf_trace_printk: client_random: 66 b7 fc
connection-38592 [003] d... 410734.672169: bpf_trace_printk: master_key: 69 89 0
可以发现最后，拿到了client_random。
以下是ecapture的日志：
2024-08-10T19:50:35-04:00 INF AppName="eCapture(旁观者)"
2024-08-10T19:50:35-04:00 INF HomePage=https://ecapture.cc
2024-08-10T19:50:35-04:00 INF Repository=https://github.com/gojue/ecapture
2024-08-10T19:50:35-04:00 INF Author="CFC4N <cfc4ncs@gmail.com>"
2024-08-10T19:50:35-04:00 INF Description="Capturing SSL/TLS plaintext without a CA certificate using eBPF. Supported on Linux/Android kernels for amd64/arm64."
2024-08-10T19:50:35-04:00 INF Version=linux_amd64:0.8.4-20240709-7e1ad52:5.15.0-117-generic
2024-08-10T19:50:35-04:00 WRN ========== module starting. ==========
2024-08-10T19:50:35-04:00 INF Kernel Info=4.18.0 Pid=42473
2024-08-10T19:50:35-04:00 INF listen=localhost:28256
2024-08-10T19:50:35-04:00 WRN Your environment is like a container. We won't be able to detect the BTF configuration.
If eCapture fails to run, try specifying the BTF mode. use `-b 2` to specify non-CORE mode.
2024-08-10T19:50:35-04:00 INF https server starting...You can update the configuration file via the HTTP interface.
2024-08-10T19:50:35-04:00 INF BTF bytecode mode: CORE. btfMode=0
2024-08-10T19:50:35-04:00 INF master key keylogger has been set. eBPFProgramType=KeyLog keylogger=ecapture_openssl_key.og
2024-08-10T19:50:35-04:00 INF module initialization. isReload=false moduleName=EBPFProbeOPENSSL
2024-08-10T19:50:35-04:00 INF Module.Run()
2024-08-10T19:50:35-04:00 INF OpenSSL/BoringSSL version found origin versionKey="OpenSSL 1.1.1k" versionKeyLower="openssl 1.1.1k"
2024-08-10T19:50:35-04:00 INF HOOK type:Openssl elf ElfType=2 binrayPath=/lib64/libssl.so.1.1 masterHookFuncs=["SSL_get_wbio","SSL_in_before","SSL_do_handshake"]
2024-08-10T19:50:35-04:00 INF setupManagers eBPFProgramType=KeyLog
2024-08-10T19:50:35-04:00 INF BPF bytecode file is matched. bpfFileName=user/bytecode/openssl_1_1_1j_kern_core_less52.o
2024-08-10T19:50:35-04:00 INF perfEventReader created mapSize(MB)=4
2024-08-10T19:50:35-04:00 INF module started successfully. isReload=false moduleName=EBPFProbeOPENSSL
2024-08-10T19:50:55-04:00 INF CLIENT_RANDOM save success CLientRandom=66b7fcdf2479d7fc638418f4ca04ee63a61d3d119763aa4fc91f28db4c722d50 TlsVersion=TLS1_2_VERSION bytes=176 eBPFProgramType=KeyLog
cfc4n commented
具体来说不仅限于mysqld，而是作为openssl服务端的问题。在gdb跟踪调试了openssl的SSL_accept函数后发现，只有在SSL_do_handshake函数返回后SSL参数才有正确的赋值。随后将openssl_masterkey.h里的"uprobe/SSL_write_key"调整成"uretprobe/SSL_write_key"，验证成功。
看上去，SSL_do_handshake函数不是准确的HOOK点，有时候TLS握手还没完成。应该是选择一个更合适的HOOK函数，对吗？
BoltzmannWXD commented
是的
cfc4n commented
hi, @BoltzmannWXD 还有后续吗？