Fail to run eBPF example (minimal) of libbpf-bootstrap on linux 6.1.15, HS38 SMP

Question

Fail to run eBPF example (minimal) of libbpf-bootstrap on linux 6.1.15, HS38 SMP

alexhsu-petaio opened this issue a year ago · 13 comments

Hi,
I run minimal eBPF program of libbpf-bootstrap and got "failed to load object 'minimal_bpf'" error. Do you have any idea about that?
Here is my environment:

buildroot: "buildroot-2022.11.1"
toolchain: arc_gnu_2022.09_prebuilt_glibc_le_archs_linux_install
linux version: 6.1.15
arc core: hs38 smp
libbpf-bootstrap commit id: d53d1201

Following is the fail log:

./minimal

libbpf: loading object 'minimal_bpf' from buffer
libbpf: elf: section(3) tp/syscalls/sys_enter_write, size 104, link 0, flags 6, type=1
libbpf: sec 'tp/syscalls/sys_enter_write': found program 'handle_tp' at insn offset 0 (0 bytes), code size 13 insns (104 bytes)
libbpf: elf: section(4) .reltp/syscalls/sys_enter_write, size 32, link 13, flags 40, type=9
libbpf: elf: section(5) license, size 13, link 0, flags 3, type=1
libbpf: license of minimal_bpf is Dual BSD/GPL
libbpf: elf: section(6) .bss, size 4, link 0, flags 3, type=8
libbpf: elf: section(7) .rodata, size 28, link 0, flags 2, type=1
libbpf: elf: section(8) .BTF, size 684, link 0, flags 0, type=1
libbpf: elf: section(10) .BTF.ext, size 160, link 0, flags 0, type=1
libbpf: elf: section(13) .symtab, size 192, link 1, flags 0, type=2
libbpf: looking for externs among 8 symbols...
libbpf: collected 0 externs total
libbpf: map 'minimal_.bss' (global data): at sec_idx 6, offset 0, flags 400.
libbpf: map 0 is "minimal_.bss"
libbpf: map 'minimal_.rodata' (global data): at sec_idx 7, offset 0, flags 80.
libbpf: map 1 is "minimal_.rodata"
libbpf: sec '.reltp/syscalls/sys_enter_write': collecting relocation for section(3) 'tp/syscalls/sys_enter_write'
libbpf: sec '.reltp/syscalls/sys_enter_write': relo #0: insn #2 against 'my_pid'
libbpf: prog 'handle_tp': found data map 0 (minimal_.bss, sec 6, off 0) for insn 2
libbpf: sec '.reltp/syscalls/sys_enter_write': relo #1: insn torvalds#6 against '.rodata'
libbpf: prog 'handle_tp': found data map 1 (minimal_.rodata, sec 7, off 0) for insn 6
libbpf: map 'minimal_.bss': skipped auto-creating...
libbpf: map 'minimal_.rodata': skipped auto-creating...
libbpf: prog 'handle_tp': relo #0: poisoning insn #2 that loads map #0 'minimal_.bss'
libbpf: prog 'handle_tp': relo #1: poisoning insn torvalds#6 that loads map #1 'minimal_.rodata'
libbpf: prog 'handle_tp': BPF program load failed: Invalid argument
libbpf: prog 'handle_tp': -- BEGIN PROG LOAD LOG --
reg type unsupported for arg#0 function handle_tp#4
0: R1=ctx(off=0,imm=0) R10=fp0
; int pid = bpf_get_current_pid_tgid() >> 32;
0: (85) call bpf_get_current_pid_tgid#14 ; R0_w=scalar()
; int pid = bpf_get_current_pid_tgid() >> 32;
1: (77) r0 >>= 32 ; R0_w=scalar(umax=4294967295,var_off=(0x0; 0xffffffff))
; if (pid != my_pid)
2:
BPF map 'minimal_.bss' is referenced but wasn't created
processed 3 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
-- END PROG LOAD LOG --
libbpf: prog 'handle_tp': failed to load: -22
libbpf: failed to load object 'minimal_bpf'
libbpf: failed to load BPF skeleton 'minimal_bpf': -22
Failed to load and verify BPF skeleton

Thanks.
Alex

Answer 1 · 2023-03-17T17:00:03.000Z

Hi @alexhsu-petaio,

Could you please provide .config for your Buildroot configuration and the same configuration file for the Linux kernel?

Answer 2 · 2023-03-18T02:33:36.000Z

Hi @kolerov ,

I upload renamed config file to txt file for buildroot and linux due to upload policy ?
petaio_archs38_haps_defconfig.txt
petaio_haps_hs_smp_defconfig.txt

I also upload kernel patch for you.
0000-linux-haps.patch

I also tried minimal_legacy eBPF program of libbpf-bootstrap and it can work. I don't know why minimal eBPF program get "reg type unsupported for arg#0 function handle_tp#4" error during BTF check phase. I also encounter the same symptoms for some of our eBPF program.

My build machine is CentOS 8, 4.18.0-448.el8.x86_64.

Thanks
Alex

Answer 3 · 2023-03-18T09:03:58.000Z

@alexhsu-petaio, could you please explain how did you build minimal step by step? Also, could you send a minimal.bpf.o binary?

Answer 4 · 2023-03-18T10:28:26.000Z

Hi @kolerov ,
I add libbpf-bootstrap into buildroot custom package and include this package when build linux image. Following attachment is this package and LIBBPF_BOOTSTRAP_APPS in libbpf-bootstrap.mk describe the eBPF apps want to build and install.
libbpf-bootstrap-pkg.tar.gz

Following attachment include minimal.bpf.o and minimal_legacy.bpf.o for your reference.
bpf-o.tar.gz

Thanks.
Alex

Answer 5 · 2023-03-21T17:08:45.000Z

@alexhsu-petaio Unfortunately, I cannot reproduce this error. Seems like the problem is in these warning lines:

libbpf: map 'minimal_.bss': skipped auto-creating...
libbpf: map 'minimal_.rodata': skipped auto-creating...

libbpf cannot allocate sections for global data and BPF program cannot get necessary data. As far as I know, this problem is not related to the version of the kernel above 5.2.

Lets try from another side. I've attached a binary of minimal which works on my side. Could you please check it on your side? Could you also send your minimal binary.

minimal-to-check.tar.gz

Answer 6 · 2023-03-22T09:42:13.000Z

Hi @kolerov ,
Thanks for your try. I have tried your mininal-to-check program and it's still fail. I attach the log of mininal-to-check program and binary of minimal_petaio program built by our site.
minimal_ebpf_dbg.tar.gz

We use the same config file for kernel (6.1.15) and buildroot (2022.11.1) but get different result, it's really strange. Do you have any idea about that?

Thanks.

Alex

Answer 7 · 2023-03-22T11:45:56.000Z

Hi @kolerov ,
I found libbpf seems use eBPF program of type BPF_PROG_TYPE_SOCKET_FILTER to check if target kernel support FEAT_GLOBAL_DATA. After enable network working support in target kernel even our plaftorm don't has network device, minimal program can create global data sections and run successfully.

Thanks for your help.

Alex

Answer 8 · 2023-03-22T13:18:48.000Z

@alexhsu-petaio Glad to hear! However, some BPF programs from these examples will work incorrectly. It happens because build system of this set of examples assumes that your target is the same as your host (it includes wrong vmlinux.h and uses host's Linux headers instead of toolchain's headers). I will explain how to avoid this problem soon.

Answer 9 · 2023-03-22T16:03:21.000Z

@alexhsu-petaio Do these steps to ensure that everything is built correctly. Suppose that my Buildroot directory is buildroot and a build directory resides in it: buildroot/build. I'm going to place sources for libbpf-bootstrap in build/overrides/libbpf-bootstrap.

Download the latest libbpf-bootstrap and retrieve modules:

git clone https://github.com/libbpf/libbpf-bootstrap
cd libbpf-bootstrap
git submodule update --init --recursive

Change a line in libbpf-bootstrap/examples/c/Makefile this way so we can substitute it by our own value in the future:

- CLANG_BPF_SYS_INCLUDES = $(shell $(CLANG) -v -E - </dev/null 2>&1 \
+ CLANG_BPF_SYS_INCLUDES := $(shell $(CLANG) -v -E - </dev/null 2>&1 \

Ensure that bpftool is installed on your host system. Build your vmlinux as usual. Then generate vmlinux.h from it:

bpftool --debug btf dump file vmlinux format c > vmlinux.h

Place vmlinux.h to libbpf-bootstrap/vmlinux/arc
Change lines in libbpf-bootstrap.mk for the package for Buildroot (use a corresponding path for your toolchain in CLANG_BPF_SYS_INCLUDES):

define LIBBPF_BOOTSTRAP_BUILD_CMDS
	$(TARGET_MAKE_ENV) $(TARGET_CONFIGURE_OPTS) $(MAKE)	\
-		-C $(LIBBPF_BOOTSTRAP_SRC_DIR) $(LIBBPF_BOOTSTRAP_APPS)
+		-C $(LIBBPF_BOOTSTRAP_SRC_DIR) $(LIBBPF_BOOTSTRAP_APPS) ARCH=$(ARCH) CLANG_BPF_SYS_INCLUDES="-I/tools/arc-linux-gnu/sysroot/usr/include"
endef

Create a file override in buildroot/build with this content:

LIBBPF_BOOTSTRAP_OVERRIDE_SRCDIR = build/overrides/libbpf-bootstrap

Add this option for your Buildroot configuration:

BR2_PACKAGE_OVERRIDE_FILE="build/override"

Then everything must be built correctly. If you are not sure then do these steps to check it:

make libbpf-bootstrap-dirclean
make libbpf-bootstrap V=1

If everything is ok you will see something like this (correct include path for your toolchain and correct vmlinux.h):

clang -g -O2 -target bpf -D__TARGET_ARCH_arc -I.output -I../../libbpf/include/uapi -I../../vmlinux/arc/ -I/tools/arc-linux-gnu/sysroot/usr/include -c minimal.bpf.c -o .output/minimal.bpf.o
...
/home/ykolerov/workspace/tasks/ebpf_jit_1/buildroot/build/host/bin/arc-linux-gnu-gcc -g -Wall -I.output -I../../libbpf/include/uapi -I../../vmlinux/arc/ -c uprobe.c -o .output/uprobe.o

Answer 10 · 2023-03-23T03:44:29.000Z

Hi @kolerov ,
Thanks a lot for your kindly shared information, it's helpful.
Here has one question about item 5. For CLANG_BPF_SYS_INCLUDES="-I/tools/arc-linux-gnu/sysroot/usr/include", do you mean the path of $(STAGING_DIR)/usr/include? I can't find "tools" folder in my build environment.

Thanks.
Alex

Answer 11 · 2023-03-23T09:38:33.000Z

@alexhsu-petaio Yes, it is. It would be even better since you don't have to use explicit path for your toolchain.

Answer 12 · 2023-03-24T10:30:33.000Z

@alexhsu-petaio So, can we consider this issue resolved? Does everything build and run fine?

Answer 13 · 2023-03-25T01:45:37.000Z

Hi @kolerov ,
This issue has been resolved. Thanks for your help.

Alex