Whitespace in functions on stackcollapse-bpftrace

[jonhoo]

#186 introduced stackcollapse-bpftrace, but since then, it appears the output of bpftrace has changed slightly. Specifically, a trace command like

# bpftrace -e 'profile:hz:999 /pid == 2141/ { @[ustack] = count(); }' 

yields output like this

@[
    alacritty::main::h459eee3a05af1eb1+38642
    std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h2fb226322c254680+3
    __libc_start_main+243
    0x5541f689495641d7
]: 1

Notice that each of the stack frames are indented. Since stackcollapse-bpftrace just pushes the line, the leading whitespace is also included in the resulting stack, giving an output like:

    alacritty::main::h459eee3a05af1eb1+38642;    std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h2fb226322c254680+3;    __libc_start_main+243;    0x5541f689495641d7 1

That doesn't seem right? We're porting stackcollapse-bpftrace in jonhoo/inferno#56, and are running into this, and wondered if we should fix this behavior in the Rust port, or whether the current behavior is intentional for some reason?

[brendangregg]

We're adding kstack(folded) and ustack(folded) calls to bpftrace soon (follow on from bpftrace/bpftrace#438) so it can emit folded format directly, at which point we don't need stackcollapse-bpftrace (which needs updating anyway).

[brendangregg]

BTW, at that point, a CPU flame graph should be doable as a one-liner, entirely using in-kernel summaries and no expensive dumping of samples to perf.data and post processing.

bpftrace -e 'profile:hz:49 { @[kstack(folded), ustack(folded)]++ } interval:s:30 { exit() }' | ./flamegraph.pl > out.svg

[jonhoo]

@brendangregg ah, awesome -- I'll reconsider that PR then!

[jonhoo]

@brendangregg I'm guessing you're alluding to bpftrace/bpftrace#430? That's great, especially when combined with the multi-key support from bpftrace/bpftrace#56! When those land I'll make sure to add that new command to the docs as "here's what you probably want" :) One question though: how does the multi-key bit know how to correctly combine kstack and ustack (i.e., using ;) in your example?