How to load debug symbols from separately installed `-dbgsym` packages?

netheril96 opened this issue 9 months ago · 7 comments

netheril96 commented 9 months ago

Please answer these questions before submitting your issue. Thanks!

What version of pprof are you using?

I installed with the command go install github.com/google/pprof@latest, which is likely on the commit 83162a5.

What operating system and processor architecture are you using?

Linux x86-64.

What did you do?

I'm profiling an application linked to libfuse (2, not 3). When viewing the profile, the libfuse functions are red and do not have source code attached. So I installed a debug symbol package sudo apt install libfuse2-dbgsym, and set the environment variable export PPROF_BINARY_PATH=/usr/lib/debug/.dwz/x86_64-linux-gnu/libfuse2.debug or export PPROF_BINARY_PATH=/usr/lib/debug/.dwz/x86_64-linux-gnu/ before running pprof. However, in either case, those functions are still red and have no source.

What did you expect to see?

Those functions are marked green.

What did you see instead?

Those functions are marked red.

aalexand commented 9 months ago

Does dpkg -L libfuse2-dbgsym print something like below for you?

/.
/usr
/usr/lib
/usr/lib/debug
/usr/lib/debug/.build-id
/usr/lib/debug/.build-id/1d
/usr/lib/debug/.build-id/1d/94114eae2ea76e09ba969d688f1d42fd956ba3.debug
/usr/lib/debug/.build-id/f5
/usr/lib/debug/.build-id/f5/fcc2b9f5db948c9712569fb24f71c623a64517.debug
/usr/lib/debug/.dwz
/usr/lib/debug/.dwz/x86_64-linux-gnu
/usr/lib/debug/.dwz/x86_64-linux-gnu/libfuse2.debug
/usr/share
/usr/share/doc
/usr/share/doc/libfuse2-dbgsym

Trying to make sure the file layout is a common one.

netheril96 commented 9 months ago

Exactly the same output

`--> dpkg -L libfuse2-dbgsym
/.
/usr
/usr/lib
/usr/lib/debug
/usr/lib/debug/.build-id
/usr/lib/debug/.build-id/1d
/usr/lib/debug/.build-id/1d/94114eae2ea76e09ba969d688f1d42fd956ba3.debug
/usr/lib/debug/.build-id/f5
/usr/lib/debug/.build-id/f5/fcc2b9f5db948c9712569fb24f71c623a64517.debug
/usr/lib/debug/.dwz
/usr/lib/debug/.dwz/x86_64-linux-gnu
/usr/lib/debug/.dwz/x86_64-linux-gnu/libfuse2.debug
/usr/share
/usr/share/doc
/usr/share/doc/libfuse2-dbgsym

netheril96 commented 9 months ago

Minimal reproducing example:

Download hello.c from libfuse 2.9.9 branch.
Compile it with gcc -Wall -Wl,--no-as-needed -lprofiler -g hello.c $(pkg-config fuse --cflags --libs) -o hello
Mount: mkdir mnt && CPUPROFILE=./profile ./hello mnt -f
On another terminal window, cat mnt/hello in a loop (otherwise too few samples can be collected).
View the profile in pprof, trying to set environment variables as described above
All methods are still red and have no sources

The compiled binary and profile collected are attached: example.tar.gz

aalexand commented 9 months ago

I get filenames when I open your binary and profile.

What does /bin/addr2line --version say for you?

netheril96 commented 9 months ago

`--> /bin/addr2line --version
GNU addr2line (GNU Binutils for Debian) 2.41.90.20240122
Copyright (C) 2024 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.

I don't think it is caused by addr2line because I can see the source of hello.c (after I insert some expensive operations to make it run longer).

aalexand commented 9 months ago

Do you mind running a command like strace -q -f -e trace=execve -e decode-pids=comm -e signal=none ./pprof -tags /tmp/libfuse/hello /tmp/libfuse/profile and posting the output?

netheril96 commented 9 months ago

OK, I figured out the problem by looking at the source code of pprof. It expects that the source code is living at the exact place as specified in the debug symbols, so I need to extract and rename the sources in order to see them matched together.