Does uftrace support aarch64 ? , segfault when trying to profile any application in userspace.
Akash-97 opened this issue · 13 comments
Built a custom distro where most of userspace is compiled with -pg and -finstrument-functions flags. Hardware is rpi4. Cross compiled capstone and uftrace:
root@buildroot:/opt/kmscube_uftrace# uftrace --version
uftrace v0.16 ( aarch64 dwarf python3 perf sched dynamic kernel )
root@buildroot:/opt/kmscube_uftrace# uname -a
Linux buildroot 6.6.47-v8 #1 SMP PREEMPT Fri Aug 30 10:39:00 IST 2024 aarch64 GNU/Linux
aarch64-rpi4-linux-gnu-gcc --version
aarch64-rpi4-linux-gnu-gcc (crosstool-NG 1.26.0) 13.2.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
uftrace crashes like the following:
root@buildroot:/opt/kmscube_uftrace# uftrace record -vv --force -a -e /usr/bin/kmscube
uftrace: running uftrace v0.16 ( aarch64 dwarf python3 perf sched dynamic kernel )
uftrace: checking binary /usr/bin/kmscube
uftrace: removing uftrace.data.old directory
uftrace: start do_child_exec, pid: 4612
uftrace: using libmcount.so library for tracing
uftrace: creating 1 thread(s) for recording
uftrace: parent ready, pid: 4612
uftrace: do_child_exec, before execv, pid: 4612
uftrace: start writer thread 0
^CWARN: child terminated by signal: 11: Segmentation fault
uftrace: stop writer thread 0
uftrace: /home/akash/mnt/ehd/roots/src/uftrace-0.16/cmds/record.c:1443:update_session_maps
ERROR: cannot find map files: No such file or directory
Any help or insight here would help.
The uftrace supports aarch64. Could you show us ldd /usr/bin/kmscube? It seems your compiler aarch64-rpi4-linux-gnu-gcc is a bit different from normal aarch64 compiler.
It seems your environment is Raspberry Pi 4 so can you try building your program natively on your board? If not possible because it's too big to compile natively, then it'd helpful if you try with a simple program such as tests/s-abc.c if it works fine with it.
Here is the output of ldd /usr/bin/kmscube:
Hmm.. I don't see something different.
Is there any kernel defconfig option that needs to be enabled for uftrace to function correctly ?
uftrace requires CONFIG_FUNCTION_GRAPH_TRACER=y but it's only for kernel tracing. It doesn't require special kernel configs for userspace tracing.
It seems your environment is Raspberry Pi 4 so can you try building your program natively on your board? If not possible because it's too big to compile natively, then it'd helpful if you try with a simple program such as tests/s-abc.c if it works fine with it.
Could you please read above and check it again with a simple program rather than your kmscube program? If the simple program works then your environment is fine, but the issue is only with the combination of kmscube and uftrace.
akash@raspberrypi:~ $ ls
a.out gmon.out main.c uftrace.data
akash@raspberrypi:~ $ uname -a
Linux raspberrypi 6.6.31+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64 GNU/Linux
akash@raspberrypi:~ $ gcc --version
gcc (Debian 12.2.0-14) 12.2.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
akash@raspberrypi:~ $ uftrace replay
# DURATION TID FUNCTION
1.389 us [ 12665] | __monstartup();
4.879 us [ 12665] | __cxa_atexit();
0.555 us [ 12665] | main();
[ 12665] | main() {
448.382 us [ 12665] | printf("hello world!");
478.716 us [ 12665] | } /* main */
akash@raspberrypi:~ $ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
Looks like its a toolchain specific issue. Is there any restriction on the version of the gcc/clang with which uftrace will work with ? and any idea on what option in crosstool ng will allow me to enable proper uftrace ?
Hmm.. are you saying that it worked with the old version (GCC 12) but not with the new GCC 13? Did you see any difference in the disassembly?
Built a custom distro where most of userspace is compiled with -pg and -finstrument-functions flags.
Did you put -pg and -finstrument-functions together?
No, those option shouldn't be used together. You added multiple hooking knobs, which makes uftrace confused. Please use only one of those options.
Just curious , adding both -pg and -finstrument-functions works on native compilation on gcc 11.x . How ? does the doc mention they are mutually exclusive options ? Also tried only each of the option in isolation with the cross tool ng toolchain and both failed.
We haven't even thought about using -pg and -finstrument-functions together. Maybe it'd be better gcc or clang doesn't allow both options together. IMHO, I'm not sure if there is such use cases.
I don't have ideas about the options of cross tool ng.
@honggyukim I found the bug. its not in my toolchain but rather how uftrace was compiled: --prefix was being wrongly set causing it to segfault. For future readers wishing to compile the uftrace from src please pay attention to --prefix which is usually set to /usr and install correctly using DESTDIR . Do not give value of DESTDIR to --prefix. Thank you.