pwru v1.0.9 symbol addresses not resolved
Closed this issue · 3 comments
Hi,
I downloaded the recent pwru and run in BPFire open source firewall with upstream kernel 6.10.11 stable release and kernel config meet pwru requirements:
[root@bpfire-9 ~]# grep -E 'CONFIG_FUNCTION_TRACER|CONFIG_FPROBE' /boot/config-6.10.11-ipfire
CONFIG_FUNCTION_TRACER=y
CONFIG_FPROBE=y
CONFIG_FPROBE_EVENTS=y
[root@bpfire-9 ~]# uname -a
Linux bpfire-9.localdomain 6.10.11-ipfire #1 SMP PREEMPT_DYNAMIC Mon Dec 2 17:14:06 GMT 2024 x86_64 QEMU Virtual CPU version 2.0.0 GenuineIntel GNU/Linux
pwru get symbol address (FUNC) unresolved
pwru --backend kprobe --kernel-btf /sys/kernel/btf/vmlinux 'dst port 444'
2024/12/02 10:15:57 Attaching kprobes (via kprobe)...
1736 / 1736 [----------------------------------------------------------------------------------------------------------] 100.00% 400 p/s
2024/12/02 10:16:01 Attached (ignored 114)
2024/12/02 10:16:01 Listening for events..
SKB CPU PROCESS NETNS MARK/x IFACE PROTO MTU LEN TUPLE FUNC
0xffff8bb7038b6900 3 /bin/pwru:4526 4026531840 0 red0:2 0x0800 1500 64 10.0.0.232:49460->10.0.0.199:444(tcp) 0xffffffff8c73d1b4
0xffff8bb7038b6900 3 /bin/pwru:4526 4026531840 0 red0:2 0x0800 1500 64 10.0.0.232:49460->10.0.0.199:444(tcp) 0xffffffff8c73cc24
0xffff8bb7038b6900 3 /bin/pwru:4526 4026531840 0 red0:2 0x0800 1500 64 10.0.0.232:49460->10.0.0.199:444(tcp) 0xffffffff8c8289d4
I can't run in multi-kprobe mode neither
[root@bpfire-9 ~]# pwru 'dst port 444'
2024/12/02 10:24:23 Attaching kprobes (via kprobe-multi)...
2024/12/02 10:24:23 Opening kprobe-multi: invalid argument (missing kernel symbol or prog's AttachType not AttachTraceKprobeMulti?)
debugfs is mounted
[root@bpfire-9 ~]# mount | grep debugfs
none on /sys/kernel/debug type debugfs (rw,relatime)
available_filter_functions available
[root@bpfire-9 ~]# grep 'skb' /sys/kernel/debug/tracing/available_filter_functions | head -10
security_sock_rcv_skb
ipv4_skb_to_auditdata
ipv6_skb_to_auditdata
vt_do_kdskbmode
vt_do_kdskbmeta
cn_rx_skb
skb_page_frag_refill
skb_orphan_partial
skb_set_owner_w
sock_alloc_send_pskb
Am I missing something ?
to trouble shoot the Opening kprobe-multi: invalid argument (missing kernel symbol or prog's AttachType not AttachTraceKprobeMulti?) I did strace -s8092 -f -o /tmp/pwru.txt pwru 'dst port 444', then found following in strace
6213 <... perf_event_open resumed>{type=0x8 /* PERF_TYPE_??? */, size=PERF_ATTR_SIZE_VER1, config=0, sample_period=0, sample_type=0, read_format=0, precise_ip=0 /* arbitrary skid */, ...}, -1, 0, -1, PERF_FLAG_FD_CLOEXEC) = -1 EINVAL (Invalid argument)
seems related to perf, and I recall in BPFire I had
commit 2cf44838bfd41c2eac98cd0d8c6bb842aea7ae4f
Author: Vincent Li <vincent.mc.li@gmail.com>
Date: Mon Sep 23 23:41:10 2024 +0000
lfs/linux: install perf tool from linux source
compile and install perf tool from linux
source for performance monitoring.
change the setting before run perf
echo -1 > /proc/sys/kernel/perf_event_paranoid
echo 0 > /proc/sys/kernel/kptr_restrict
so I did echo -1 > /proc/sys/kernel/perf_event_paranoid; echo 0 > /proc/sys/kernel/kptr_restrict, then it works :)
[root@bpfire-9 ~]# echo -1 > /proc/sys/kernel/perf_event_paranoid
[root@bpfire-9 ~]# echo 0 > /proc/sys/kernel/kptr_restrict
[root@bpfire-9 ~]# perf top
[root@bpfire-9 ~]# pwru 'dst port 444'
2024/12/02 18:32:27 Attaching kprobes (via kprobe-multi)...
1623 / 1623 [--------------------------------------------------------------------------------------------------------------------------------] 100.00% ? p/s
2024/12/02 18:32:27 Attached (ignored 0)
2024/12/02 18:32:27 Listening for events..
SKB CPU PROCESS NETNS MARK/x IFACE PROTO MTU LEN TUPLE FUNC
0xffff8bb70c295500 3 <empty>:0 4026531840 0 red0:2 0x0800 1500 64 10.0.0.232:61715->10.0.0.199:444(tcp) inet_gro_receive
0xffff8bb70c295500 3 <empty>:0 4026531840 0 red0:2 0x0800 1500 64 10.0.0.232:61715->10.0.0.199:444(tcp) tcp4_gro_receive
0xffff8bb70c295500 3 <empty>:0 4026531840 0 red0:2 0x0800 1500 64 10.0.0.232:61715->10.0.0.199:444(tcp) tcp_gro_pull_header
0xffff8bb70c295500 3 <empty>:0 4026531840 0 red0:2 0x0800 1500 64 10.0.0.232:61715->10.0.0.199:444(tcp) tcp_gro_receive
0xffff8bb70c295500 3 <empty>:0 4026531840 0 red0:2 0x0800 1500 64 10.0.0.232:61715->10.0.0.199:444(tcp) packet_rcv
0xffff8bb70c295500 3 <empty>:0 4026531840 0 red0:2 0x0800 1500 64 10.0.0.232:61715->10.0.0.199:444(tcp) skb_push
Could you open a PR to add a warning message when missing kernel symbol? i.e. aforementioned --backend kprobe case.
If I remember correctly, --backend kprobe relies on /proc/kallsyms while --backend kprobe-multi counts on /sys/kernel/debug/tracing/available_filter_functions. That must be the reason you see different results in different backend.