Fix build issue that `struct user_pt_regs` is not found on aarch64
Closed this issue · 5 comments
A compile issue that it is unable to find the definition of struct user_pt_regs when compiling scx_rlfifo and scx_rustland (they use BPF_KPROBE and thus requires user_pt_regs).
struct user_pt_regs indeed is not included in scheds/include/vmlinux/vmlinux.h.
Regenerating vmlinux.h from the linux upstream 88264981f20 can solve the problem.
This is likely an issue only on aarch64. Generating vmlinux.h during compilation, as bpftool repo does, may be a solution.
Log:
:~/scx$ cargo build
warning: /home/debian/scx/rust/scx_rustland_core/Cargo.toml: unused manifest key: lib.include
Compiling scx_rustland_core v2.1.2 (/home/debian/scx/rust/scx_rustland_core)
Compiling scx_stats_derive v1.0.5 (/home/debian/scx/rust/scx_stats/scx_stats_derive)
Compiling scx_bpfland v1.0.5 (/home/debian/scx/scheds/rust/scx_bpfland)
Compiling scx_lavd v1.0.5 (/home/debian/scx/scheds/rust/scx_lavd)
Compiling scx_rusty v1.0.5 (/home/debian/scx/scheds/rust/scx_rusty)
Compiling scx_loader v1.0.5 (/home/debian/scx/rust/scx_loader)
Compiling scx_rustland v1.0.5 (/home/debian/scx/scheds/rust/scx_rustland)
Compiling scx_rlfifo v1.0.5 (/home/debian/scx/scheds/rust/scx_rlfifo)
error: failed to run custom build command for `scx_rlfifo v1.0.5 (/home/debian/scx/scheds/rust/scx_rlfifo)`
Caused by:
process didn't exit successfully: `/home/debian/scx/target/debug/build/scx_rlfifo-1aec066430c5fe0f/build-script-build` (exit status: 101)
--- stdout
scx_utils:clang=("clang", "17.0.6", "aarch64") ["-g", "-O2", "-Wall", "-Wno-compare-distinct-pointer-types", "-D__TARGET_ARCH_arm64", "-mcpu=v3", "-mlittle-endian", "-idirafter", "/usr/lib/llvm-17/lib/clang/17/include", "-idirafter", "/usr/local/include", "-idirafter", "/usr/include/aarch64-linux-gnu", "-idirafter", "/usr/include", "-I/home/debian/scx/target/debug/build
/scx_rlfifo-f75931752899ae49/out/scx_utils-bpf_h", "-I/home/debian/scx/target/debug/build/scx_rlfifo-f75931752899ae49/out/scx_utils-bpf_h/vmlinux", "-I/home/debian/scx/target/debug/build/scx_rlfifo-f75931752899ae49/out/scx_utils-bpf_h/bpf-compat"]
cargo:rerun-if-env-changed=TARGET
cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64-unknown-linux-gnu
cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS_aarch64_unknown_linux_gnu
cargo:rerun-if-env-changed=BINDGEN_EXTRA_CLANG_ARGS
cargo:rerun-if-changed=/usr/lib/llvm-17/lib/clang/17/include/stdbool.h
--- stderr
thread 'main' panicked at scheds/rust/scx_rlfifo/build.rs:8:10:
called `Result::unwrap()` on an `Err` value: failed to build `main.bpf.c`
Caused by:
0: Failed to compile /home/debian/scx/target/debug/build/scx_rlfifo-f75931752899ae49/out/bpf.bpf.o from main.bpf.c
1: Command `clang -g -O2 -Wall -Wno-compare-distinct-pointer-types -D__TARGET_ARCH_arm64 -mcpu=v3 -mlittle-endian -idirafter /usr/lib/llvm-17/lib/clang/17/include -idirafter /usr/local/include -idirafter /usr/include/aarch64-linux-gnu -idirafter /usr/include -I/home/debian/scx/target/debug/build/scx_rlfifo-f75931752899ae49/out/scx_utils-bpf_h -I/home/debian/scx/targ
et/debug/build/scx_rlfifo-f75931752899ae49/out/scx_utils-bpf_h/vmlinux -I/home/debian/scx/target/debug/build/scx_rlfifo-f75931752899ae49/out/scx_utils-bpf_h/bpf-compat -I /tmp/.tmpjZKrar/bpf/src -fno-stack-protector -g -O2 -target bpf -c main.bpf.c -o /home/debian/scx/target/debug/build/scx_rlfifo-f75931752899ae49/out/bpf.bpf.o` failed (exit status: 1)
2: main.bpf.c:284:5: error: incomplete definition of type 'struct user_pt_regs'
284 | int BPF_KPROBE(kprobe_handle_mm_fault, void *vma,
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
285 | unsigned long address, unsigned int flags)
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/tmp/.tmpjZKrar/bpf/src/bpf/bpf_tracing.h:817:20: note: expanded from macro 'BPF_KPROBE'
817 | return ____##name(___bpf_kprobe_args(args)); \
| ^~~~~~~~~~~~~~~~~~~~~~~~ environment:
:~/scx$ uname -a
Linux debian 6.11.0+ #1 SMP Sat Sep 21 18:02:59 PDT 2024 aarch64 GNU/Linux
:~/scx$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 12 (bookworm)
Release: 12
Codename: bookwormSo, I thought a bit about it and it seems like we just need to have per-arch vmlinux.h shipped in scx_utils. @minosfuture, do you wanna take a stab at it?
@htejun, sure thing. But these files are quite large though. Do you mind clarifying why not to follow how bpftool generates vmlinux.h on the fly?
For a couple reasons:
- It simplifies build process and makes rust projects self contained (e.g. you can do
cargo install scx_layeredand it won't need any external dependencies outside the toolchain). vmlinux.hshould track the latest regardless of the host that schedulers are being built on. e.g. backward compatibility macros in underinclude/scxassume all the latest definitions.
These two scripts might help you @minosfuture
This one, when ran from that directory (i.e. it needs the dockerfile/other script there) will generate a vmlinux.h from a commit hash, kernel git repo, and branch name:
https://github.com/sched-ext/scx-backports/blob/35aacc4-1.0.5-d-fixes/backport-scripts/generate-backport-vmlinux.sh
In the event you don't want to deal with that dockerfile (cloning/building the kernel in docker while iterating is... painful), you can use the commands towards the end of this script to generate a vmlinux.h from a vmlinux:
@htejun @likewhatevs I submitted a PR #793 for this. Please review, thanks!
I can also add another commit to include a simple script (similar to what @likewhatevs had) into the repo, if that'll be useful.