RFE: trace TLS in container
vincentmli opened this issue · 28 comments
Hi,
for example I have a netshoot pod running in kubernetes, when I run curl
kubectl exec -it netshoot-hostnetwork -- curl -k -v https://10.1.34.88/
I want to trace the TLS connection from the curl from the netshoot-hostnetwork pod, the curl in netshoot-hostnetwork pod has libssl below in the pod namespace
kubectl exec -it netshoot-hostnetwork -- ldd /usr/bin/curl
/lib/ld-musl-x86_64.so.1 (0x7f932eda0000)
libcurl.so.4 => /usr/lib/libcurl.so.4 (0x7f932ece5000)
libz.so.1 => /lib/libz.so.1 (0x7f932eccb000)
libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f932eda0000)
libnghttp2.so.14 => /usr/lib/libnghttp2.so.14 (0x7f932eca6000)
libssl.so.1.1 => /lib/libssl.so.1.1 (0x7f932ec25000)
libcrypto.so.1.1 => /lib/libcrypto.so.1.1 (0x7f932e9a4000)
libbrotlidec.so.1 => /usr/lib/libbrotlidec.so.1 (0x7f932e998000)
libbrotlicommon.so.1 => /usr/lib/libbrotlicommon.so.1 (0x7f932e975000)
is this possible? or in general, could we improve ecapture to capture container TLS traffic?
I could do this:
find the netshoot docker ID
docker ps | grep -w netshoot
b37ffd7a8341 nicolaka/netshoot "/bin/sleep 3600" 50 minutes ago Up 50 minutes k8s_netshoot_netshoot-hostnetwork_default_751666e9-5a23-4da1-952f-379288b47f97_0
docker inspect the ID
docker inspect b37ffd7a8341 | grep '"MergedDir"'
"MergedDir": "/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged",
find libssl
find /var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged -name "libssl*"
/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged/lib/libssl.so.1.1
/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged/usr/lib/libssl.so.1.1
use ecapture with the correct libssl path
ecapture tls --libssl="/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged/lib/libssl.so.1.1" --hex
2022/05/30 14:02:27 pid info :2825032
2022/05/30 14:02:27 start to run EBPFProbeOPENSSL module
2022/05/30 14:02:27 start to run EBPFProbeGNUTLS module
2022/05/30 14:02:27 HOOK type:2, binrayPath:/var/lib/docker/overlay2/02c5fe50b9c6a817c47117ebddd8be82cf4095a6ff278f197519b1cedb7c3d75/merged/lib/libssl.so.1.1
2022/05/30 14:02:27 libPthread so Path:/lib64/libpthread.so.0
2022/05/30 14:02:27 target all process.
2022/05/30 14:02:27 start to run EBPFProbeNSPR module
2022/05/30 14:02:27 stat /usr/lib/libnspr4.so: no such file or directory
2022/05/30 14:02:27 HOOK type:2, binrayPath:/lib64/libgnutls.so.30
2022/05/30 14:02:27 target all process.
execute the curl from netshoot pod
kubectl exec -it netshoot-hostnetwork -- curl -k -v https://10.1.34.88/
* Trying 10.1.34.88:443...
* Connected to 10.1.34.88 (10.1.34.88) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
* subject: C=US; ST=WA; L=Seattle; O=MyCompany; OU=IT; CN=localhost.localdomain; emailAddress=root@localhost.localdomain
* start date: Oct 1 19:29:04 2020 GMT
* expire date: Sep 29 19:29:04 2030 GMT
* issuer: C=US; ST=WA; L=Seattle; O=MyCompany; OU=IT; CN=localhost.localdomain; emailAddress=root@localhost.localdomain
* SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET / HTTP/1.1
> Host: 10.1.34.88
> User-Agent: curl/7.83.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Server: BigIP
* HTTP/1.0 connection set to keep alive
< Connection: Keep-Alive
< Content-Length: 10
<
* Connection #0 to host 10.1.34.88 left intact
IT WORKS
the ecapture output:
2022/05/30 14:03:34 PID:2825502, Comm:curl, TID:2825502, Version:TLS1_2_VERSION, Send 74 bytes to [ADDR_NOT_FOUND], Payload:
GET / HTTP/1.1
Host: 10.1.34.88
User-Agent: curl/7.83.1
Accept: */*
2022/05/30 14:03:34 PID:2825502, Comm:curl, TID:2825502, Version:TLS1_2_VERSION, Recived 88 bytes from [ADDR_NOT_FOUND], Payload:
HTTP/1.0 200 OK
Server: BigIP
Connection: Keep-Alive
Content-Length: 10
IT WORKS
yes, you are right. eh....Is your problem solved?
yes it is resolved, thanks for this great project :)
yes it is resolved, thanks for this great project :)
Thanks the test.
Can we have a sample to use this tool into the container?
hm, this brings me thoughts about multi tenant cloud environment, that someone could run ecapture as container and gain privilege on the node and sniff other tenant containers TLS connection, is that possible ?
Of course, if the container is authorized with the SYS_ADMIN permission, then it can obtain the communication plaintext of all networks on this host.
So, That is an other topic about eBPF security on runtime.
ref: https://github.com/ehids/ebpf-slide/blob/master/security/us-21-With-Friends-Like-EBPF-Who-Needs-Enemies.pdf
demo: https://www.youtube.com/watch?v=zgsHc8apFGs
posts on chinese: https://www.cnxct.com/evil-use-ebpf-and-how-to-detect-ebpf-rootkit-in-linux/?f=ecapture-github
I wonder if https://github.com/cilium/tetragon is able to detect this scenario, I assume your ehids could detect this too
tetragon is the learning objective of ehids.
I am not familiar with docker, can we create a docker file for ecapture to run ecapture in container/pod ?
ok, good, just one more question, is it possible to give --libssl argument multiple libssl path so ecapture could capture multiple TLS connections for multiple processes/commands ? I guess we can run multiple ecapture commands to achieve that, just wonder :)
1, eCapture can capture all process TLS paintext who use the same libssl.so default. and can use pid arg to filter.
2, Can run multiple eCapture with every libssl path to achieve that.
3, Of course, an eCapture process can hook multiple ssl/tls SO file, but I don't think this is a necessary requirement.
ah, right I got 1), I am thinking in k8s pod/container scenario that each pod/container might have their own libssl copy, yes 2) can achieve that.
by the way, I recorded a short video playing with ecapture https://youtu.be/Au1YeB0nz3g
Good job. I'll create User Manual WIKI tomorrow with your video, thanks. 😊
Hi @cfc4n sorry to bother you again, I added the ecapture binary in netshoot pod like vincentmli/netshoot@72633ba so I can run ecapture from netshoot pod in k8s, the netshoot pod yaml file has privilege permission like below
apiVersion: v1
kind: Pod
metadata:
name: netshoot-ecap
spec:
hostPID: true
hostNetwork: true
nodeSelector:
dedicated: master
containers:
- name: netshoot-ecap
image: vli39/netshoot:ecap
command:
- /bin/sleep
- "3600"
volumeMounts:
- mountPath: /mnt
name: host-slash
securityContext:
privileged: true
volumes:
- name: host-slash
hostPath:
path: /
type: ''
you can see I mounted the node root / in netshoot pod /mnt to access the host libssl, but it errors out with "lstat /etc/ld.so.conf: no such file or directory", I wonder why ecapture is trying to check host /etc/ld.so.conf, ecapture is statically build, I suspect the host libssl requiring host /etc/ld.so.conf, but I can't mount host /etc to netshoot pod /etc, wonder if you have better idea :)
kubectl exec -it netshoot-ecap -- /bin/bash
bash-5.1# ls -l /mnt/lib64/libssl*
lrwxrwxrwx 1 root root 16 Mar 30 2021 /mnt/lib64/libssl.so -> libssl.so.1.1.1g
lrwxrwxrwx 1 root root 16 Mar 30 2021 /mnt/lib64/libssl.so.1.1 -> libssl.so.1.1.1g
-rwxr-xr-x 1 root root 615576 Mar 30 2021 /mnt/lib64/libssl.so.1.1.1g
-rwxr-xr-x 1 root root 394632 Dec 18 2020 /mnt/lib64/libssl3.so
bash-5.1# ecapture tls --libssl="/mnt/lib64/libssl.so.1.1.1g" --hex
2022/06/03 14:51:29 pid info :1391071
2022/06/03 14:51:29 start to run EBPFProbeOPENSSL module
2022/06/03 14:51:29 lstat /etc/ld.so.conf: no such file or directory
2022/06/03 14:51:29 invalid argument
eh, eCapture will find pthread.so to capture IP ADDRESS default , and you can set pthread.so argument with pthread falg.
ecapture is statically built
ldd /usr/local/bin/ecapture
not a dynamic executable
but are you saying eCapture dynamically linked with pthread.so? how do I specify ecapture with pthread flag? sorry I am not following you :)
eh, eCapture will find
pthread.soto captureIP ADDRESSdefault , and you can setpthread.soargument withpthreadfalg.
bash-5.1# ecapture tls --libssl="/mnt/var/lib/docker/overlay2/9ee7c714e2a3435b7cdcc81cc595afa2d9bb42136439bda35677c85bbf7d160c/merged/usr/lib/x86_64-linux-gnu/libssl.so.1.1" --pthread="/mnt/var/lib/docker/overlay2/9ee7c714e2a3435b7cdcc81cc595afa2d9bb42136439bda35677c85bbf7d160c/merged/usr/lib/x86_64-linux-gnu/libpthread.so.0"
2022/06/05 04:36:54 pid info :10668
2022/06/05 04:36:54 start to run EBPFProbeOPENSSL module
2022/06/05 04:36:54 start to run EBPFProbeGNUTLS module
2022/06/05 04:36:54 invalid argument
bash-5.1#
bash-5.1# ecapture tls --libssl="/mnt/var/lib/docker/overlay2/9ee7c714e2a3435b7cdcc81cc595afa2d9bb42136439bda35677c85bbf7d160c/merged/usr/lib/x86_64-linux-gnu/libssl.so.1.1" --pthread="/mnt/var/lib/docker/overlay2/9ee7c714e2a3435b7cdcc81cc595afa2d9bb42136439bda35677c85bbf7d160c/merged/usr/lib/x86_64-linux-gnu/libpthread.so.0" -h
NAME:
tls - alias name:openssl , use to capture tls/ssl text content without CA cert.
USAGE:
ecapture tls [flags]
DESCRIPTION:
use eBPF uprobe to capture process event data, not used libpcap.
Can used to trace, debug, database audit, security event aduit etc.
OPTIONS:
--curl="" curl or wget file path, use to dectet openssl.so path, default:/usr/bin/curl
--firefox="" firefox file path, default: /usr/lib/firefox/firefox.
--gnutls="" libgnutls.so file path, will automatically find it from curl default.
-h, --help[=false] help for tls
--libssl="" libssl.so file path, will automatically find it from curl default.
--nspr="" libnspr44.so file path, will automatically find it from curl default.
--pthread="" libpthread.so file path, use to hook connect to capture socket FD.will automatically find it from curl.
--wget="" wget file path, default: /usr/bin/wget.
GLOBAL OPTIONS:
-d, --debug[=false] enable debug logging
--hex[=false] print byte strings as hex encoded strings
-p, --pid=0 if pid is 0 then we target all pids
bash-5.1#
ecapture is statically built
ldd /usr/local/bin/ecapture not a dynamic executablebut are you saying eCapture dynamically linked with
pthread.so? how do I specify ecapture withpthreadflag? sorry I am not following you :)
sure, ecapture is statically built . but eCapture hook connect()function to capture IP ADDRESS ,sometimes connect function was define inpthread.so. so need to search the path.
sometimes , defined in /lib64/libc.so.6 , you can use ldd which curl|grep libc.so|awk '{print $3}'|xargs objdump -T |grep connect to confirm it.
[root@VM-16-13-centos bin]# ldd `which curl`|grep libc.so|awk '{print $3}'|xargs objdump -T |grep connect
0000000000112f90 w DF .text 000000000000009b GLIBC_2.2.5 __connect
0000000000112f90 w DF .text 000000000000009b GLIBC_2.2.5 connect@BurlyLuo @vincentmli
It will skip directory search if lib path args was set.
ah, sorry I missed the output libPthread so Path:/lib64/libpthread.so.0 when run ecapture normally
ecapture tls --libssl=/lib64/libssl.so.1.1.1g
2022/06/05 10:53:59 pid info :3232826
2022/06/05 10:53:59 start to run EBPFProbeOPENSSL module
2022/06/05 10:53:59 start to run EBPFProbeGNUTLS module
2022/06/05 10:53:59 stat /usr/lib/libgnutls.so.30: no such file or directory
2022/06/05 10:53:59 start to run EBPFProbeNSPR module
2022/06/05 10:53:59 stat /usr/lib/libnspr4.so: no such file or directory
2022/06/05 10:53:59 HOOK type:2, binrayPath:/lib64/libssl.so.1.1.1g
2022/06/05 10:53:59 libPthread so Path:/lib64/libpthread.so.0 <====MISSED THIS
2022/06/05 10:53:59 target all process.
but still error when I run ecapture in container with --pthread=/mnt/lib64/libpthread.so.0, it looks to me when putting ecapture in container, for some reason, ecapture could not find the correct pthread lib even with --pthread argument, and still try to locate /etc/ld.so.conf
kubectl exec -it netshoot-ecap -- /bin/bash
bash-5.1#
bash-5.1# ls -l /mnt/lib64/libpth*
-rwxr-xr-x 1 root root 320504 Jul 20 2020 /mnt/lib64/libpthread-2.28.so
lrwxrwxrwx 1 root root 27 Jul 20 2020 /mnt/lib64/libpthread.so -> ../../lib64/libpthread.so.0
lrwxrwxrwx 1 root root 18 Jul 20 2020 /mnt/lib64/libpthread.so.0 -> libpthread-2.28.so
bash-5.1# ecapture tls --libssl=/mnt/lib64/libssl.so.1.1.1g --pthread=/mnt/lib64/libpthread.so.0
2022/06/05 15:01:40 pid info :3237812
2022/06/05 15:01:40 start to run EBPFProbeOPENSSL module
2022/06/05 15:01:40 start to run EBPFProbeGNUTLS module
2022/06/05 15:01:40 lstat /etc/ld.so.conf: no such file or directory
2022/06/05 15:01:40 invalid argument
tried create /etc/ld.so.conf with mounted host / in container, still can't find connect I guess
bash-5.1# echo "/mnt/lib64" > /etc/ld.so.conf
bash-5.1# cat /etc/ld.so.conf
/mnt/lib64
bash-5.1# ecapture tls --libssl="/mnt/var/lib/docker/overlay2/16e8490ca25fe4c685825f60ed33d704ebf0a1689b9887c24d0b98b12ae961e2/merged/usr/lib64/libssl.so.1.0.2k" --hex
2022/06/05 15:32:04 pid info :3257576
2022/06/05 15:32:04 start to run EBPFProbeOPENSSL module
2022/06/05 15:32:04 start to run EBPFProbeGNUTLS module
2022/06/05 15:32:04 invalid argument
bash-5.1# ecapture tls --pthread="/mnt/var/lib/docker/overlay2/16e8490ca25fe4c685825f60ed33d704ebf0a1689b9887c24d0b98b12ae961e2/merged/usr/lib64/libpthread.so.0" --libssl="/mnt/var/lib/docker/overlay2/16e8490ca25fe4c685825f60ed33d704ebf0a1689b9887c24d0b98b12ae961e2/merged/usr/lib64/libssl.so.1.0.2k" --hex
2022/06/05 15:29:01 pid info :3255607
2022/06/05 15:29:01 start to run EBPFProbeOPENSSL module
2022/06/05 15:29:01 start to run EBPFProbeGNUTLS module
2022/06/05 15:29:01 invalid argument
may be it's a bug. can you send a new issue for it?
sure, will do, appreciate the help!
opened #69
close this since question is answered by @cfc4n