kprobes may persist after program exit

[brayniac]

With the changes in #43 we now have a minimum repro for a case I've noticed in practical use of this library. In short, it seems there's some issue in the Drop implementations that results in kprobes remaining registered in debugfs after program exit.

This issue is specific to kernels prior to 4.17 where ebpf programs are opened using perf event ABI instead of through debugfs. It is unclear if a similar issue exists in newer kernels.

However, for kernels like 4.9 and 4.14 we can easily reproduce this issue using the opensnoop example.

Steps to reproduce:

# the following command should show no registered kprobes
sudo cat /sys/kernel/debug/tracing/kprobe_events
# run the example and ctrl-c after some time, will see EBUSY on `write()`
sudo ./target/release/examples/opensnoop
# show the registered kprobes again, note they persist after program exit
sudo cat /sys/kernel/debug/tracing/kprobe_events

Error messages look like this:

write(-:kprobes/r_do_sys_open_bcc_16070): Device or resource busy
write(-:kprobes/p_do_sys_open_bcc_16070): Device or resource busy

Looking at the kernel source for 4.14.128, I see two cases where EBUSY could be returned:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/kernel/trace/trace_kprobe.c?h=v4.14.128#n524

We need to see if there's more we need to do as part of the Drop implementations to ensure the probes are cleaned up properly.

[waynr]

Is it possible to tie a kprobe to the process that instantiates it? I can imagine use cases where one might want kprobes to exist beyond the lifetime of the instantiating process, but it doesn't seem like a safe default. Especially for an example bpf program.

I am interested in taking this on if no one else is looking into it.

[brayniac]

@waynr - I'm not sure it'd be useful to purposefully leave kprobes around after process exit. You'd need to come up with a way for a new process to access the data. It's unclear to me what the utility would be here.

I'd welcome a fix to this library that will result in the kprobes being properly cleaned up during Drop on the kernels mentioned in the main description

[lukaspustina]

Hi, I'm observing the same issue. Any news on this?

[brayniac]

Hi, I'm observing the same issue. Any news on this?

@lukaspustina - what distro, kernel, and bcc versions?

[lukaspustina]

root@bionic64-vagrant-vm:/# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"
root@bionic64-vagrant-vm:/# uname -a
Linux bionic64-vagrant-vm 4.15.0-106-generic #107-Ubuntu SMP Thu Jun 4 11:27:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@bionic64-vagrant-vm:/# bpftrace --version
bpftrace v0.10.0-72-gc4c4

Im using the bcc crate in version "0.0.17" which pulls bcc-sys in version "0.13.0".

Thanks for your support.