rr-debugger/rr

Will RR work in WSL 2

dulinriley opened this issue · 14 comments

dulinriley commented

I want to try out RR on a project of mine, and the easiest way for me to try it out is through a WSL 2 Ubuntu VM on a Windows computer.

After downloading v5.3.0 using the instructions from the homepage:

wget https://github.com/mozilla/rr/releases/download/5.3.0/rr-5.3.0-Linux-$(uname -m).deb
sudo dpkg -i rr-5.3.0-Linux-$(uname -m).deb

And using it on an example binary:

rr record ./a.out

I get the following error message:

[FATAL /home/roc/rr/rr/src/PerfCounters.cc:314:start_counter() errno: ENOENT] Unable to open performance counter with 'perf_event_open'; are perf events enabled? Try 'perf record'.
=== Start rr backtrace:
rr(_ZN2rr13dump_rr_stackEv+0x3b)[0x5f229b]
rr(_ZN2rr15notifying_abortEv+0x47)[0x5f2327]
rr[0x52cbea]
rr[0x52dede]
rr(_ZN2rr12PerfCounters23default_ticks_semanticsEv+0x1a)[0x52e9ca]
rr(_ZN2rr7SessionC1Ev+0x139)[0x5bc369]
rr(_ZN2rr13RecordSessionC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt6vectorIS6_SaIS6_EESD_RKNS_20DisableCPUIDFeaturesENS0_16SyscallBufferingEiNS_7BindCPUES8_PKNS_9TraceUuidE+0x4d)[0x54069d]
rr(_ZN2rr13RecordSession6createERKSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EESB_RKNS_20DisableCPUIDFeaturesENS0_16SyscallBufferingEhNS_7BindCPUERKS7_PKNS_9TraceUuidE+0xba9)[0x541769]
rr(_ZN2rr13RecordCommand3runERSt6vectorINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESaIS7_EE+0xe14)[0x535c24]
rr(main+0x353)[0x4ac743]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f77633761e3]
rr(_start+0x29)[0x4acb59]
=== End rr backtrace
Aborted

Given the advice, I looked into how to use perf record on WSL 2. According to this issue: microsoft/WSL#329, it looks like all I need to do is build perf from source (which is thankfully very easy) and run it.

Here's what I ran:

git clone --depth 1 --shallow-submodules https://github.com/microsoft/WSL2-Linux-Kernel.git
cd ./WSL2-Linux-Kernel/tools
make perf
./perf/perf record ~/a.out

This command seems to run fine! I can view the trace with perf report and there were no error messages.

So why is perf_event_open returning ENOENT to RR?

Some of my build info in case any of that is relevant:

Field Value
Windows Build number 19041.173
CPU Intel Core i7-7700HQ
uname -sr Linux 4.19.84-microsoft-standard
rr --version 5.3.0

EDIT: Tried on the newest Windows Insiders build as of 2020-04-13 (build number 19603), still no luck

dulinriley commented

I didn't add this as a comment on #2118 because that was for WSL 1, which wasn't trying to allow tools like perf to work on them.
Now WSL 2 is more like a VM (using Hyper-V), so it might be able to work.

rocallahan commented

Hyper-V doesn't virtualize hardware performance counters as far as I know. If you run perf list and it shows [Hardware event] anywhere then hardware counters work, but I expect that it won't. So this is really an issue you would need to take up with Microsoft.

dulinriley commented

I don't see any [Hardware event], but I do see a [Hardware breakpoint] and [Raw hardware event descriptor].

This article seems to imply that Hyper-V does allow perf counters: https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/manage/performance-monitoring-hardware

although WSL 2 VMs do not show up in the various *-VMProcessor powershell commands.

rocallahan commented

I don't see any [Hardware event], but I do see a [Hardware breakpoint] and [Raw hardware event descriptor].

[Hardware breakpoint] is not a performance event, it just gets exposed through perf because perf exposes lots of random stuff. [Raw hardware event descriptor] is just a generic line that is always printed. So your WSL environment doesn't have hardware performance counters enabled.

This article seems to imply that Hyper-V does allow perf counters: https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/manage/performance-monitoring-hardware

Great, that's good news! You need to find a way to turn that on for WSL2.

If you successfully turn it on you should see stuff like this in perf list:

  branch-instructions OR branches                    [Hardware event]
  branch-misses                                      [Hardware event]
  bus-cycles                                         [Hardware event]
  cache-misses                                       [Hardware event]
  cache-references                                   [Hardware event]
  cpu-cycles OR cycles                               [Hardware event]
  instructions                                       [Hardware event]
  ref-cycles                                         [Hardware event]

Then you can try rr. There are many more reasons why rr might not work, some which we could fix and some we couldn't, but it's definitely not going to work until you get that perf list output.

  • 👍3
khuey commented

I'm 99.9% sure that WSL's ptrace implementation is still not sufficient too.

rocallahan commented

I'm 99.9% sure that WSL's ptrace implementation is still not sufficient too.

That's WSL1. WSL2 is real Linux so ptrace should be OK.

kruxmeier commented

The performance counters can't be unlocked via normal Hyper-V settings, but you can modify vmwp.exe.

Trass3r commented

It will work in the Store version of WSL.

tyler274 commented

It will work in the Store version of WSL.

What do you mean?

Ah I see HW perf counters was shipped with 0.50.2

edit: It seems to work in WSL2 now.

J-jaeyoung commented

For those who are still experiencing a perf issue, just try wsl --update command on powershell and relaunch the wsl.
After doing so, rr works perfectly fine on wsl2+win11 setting.

Kodp commented

@J-jaeyoung
I'm using WSL2 and Windows 11, but I'm still encountering issues.

When I run rr record ./rtarget -q -i ./raw_5_exploit.o, it saves execution to the trace directory `/home/syh/.local/share/rr/rtarget-4', but then I receive the following fatal error:

[FATAL /home/ubuntu/rr/src/PerfCounters.cc:316:start_counter()] Unable to open performance counter with 'perf_event_open'; are hardware perf events available? See https://github.com/rr-debugger/rr/wiki/Will-rr-work-on-my-system

When I follow the instructions in rr/wiki/Will-rr-work-on-my-system and execute the command:

$ perf stat -e br_inst_retired.conditional true

I encounter the following warning:

WARNING: perf not found for kernel 5.15.146.1-microsoft

  You may need to install the following packages for this specific kernel:
    linux-tools-5.15.146.1-microsoft-standard-WSL2
    linux-cloud-tools-5.15.146.1-microsoft-standard-WSL2

  You may also want to install one of the following packages to keep up to date:
    linux-tools-standard-WSL2
    linux-cloud-tools-standard-WSL2

I tried to install the mentioned Microsoft packages using apt or snap, but they couldn't be found:

$ sudo apt install linux-tools-5.15.146.1-microsoft-standard-WSL2
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package linux-tools-5.15.146.1-microsoft-standard-WSL2
E: Couldn't find any package by glob 'linux-tools-5.15.146.1-microsoft-standard-WSL2'

Have you encountered these issues before?

J-jaeyoung commented

@song-yuhang

Try the following commands.

git clone --depth 1 --shallow-submodules https://github.com/microsoft/WSL2-Linux-Kernel.git
cd ./WSL2-Linux-Kernel/tools
make perf

cp perf /usr/local/bin     # if you want
vimkim commented 
❯ perf stat -e br_inst_retired.conditional true
event syntax error: 'br_inst_retired.conditional'
                     \___ parser error
Run 'perf list' for a list of valid events

 Usage: perf stat [<options>] [<command>]

    -e, --event <event>   event selector. use 'perf list' to list available events

This is my result, after I did what @J-jaeyoung exactly mentioned. I ended up here.

  • 👍2
GitMensch commented

@vimkim The exact name depends on both the processor and the perf version - see https://github.com/rr-debugger/rr/wiki/Will-rr-work-on-my-system