arduino/ArduinoCore-zephyr

How to decode a fault address?

KurtE opened this issue · 2 comments

KurtE commented

Suppose the code on a Nano BLE 33 Sense V2 faults and fault shows up in Terminal window, like:

[00:00:02.586,456] <err> os: ***** USAGE FAULT *****
[00:00:02.595,092] <err> os:   No coprocessor instructions
[00:00:02.604,278] <err> os: r0/a1:  0x428861e1  r1/a2:  0x40510c3c  r2/a3:  0xe6666668
[00:00:02.616,241] <err> os: r3/a4:  0x10a21878 r12/ip:  0x00000000 r14/lr:  0x2000badf
[00:00:02.628,173] <err> os:  xpsr:  0x21000000
[00:00:02.636,352] <err> os: Faulting instruction address (r15/pc): 0x2000bade
[00:00:02.647,430] <err> os: >>> ZEPHYR FATAL ERROR 33: Unknown error on CPU 0
[00:00:02.658,508] <err> os: Current thread: 0x20001898 (main)
[00:00:02.668,060] <err> os: Halting system
uart:~$

IS there a secret on how to map the fault address: 0x2000bade
to where it actually faulted in the sketch code?

If I look within the map file generated by the Arduino build: There are no symbols anywhere near this:
for example searching for 2000 will fail.

If it faulted within the code built as part of build.sh I can typically find the nearest symbol within the map file:
.../ArduinoCore-zephy/build/zephyr/zephyr.map.

In this case the closest symbols within that file shows:

noinit          0x0000000020009740    0x24a68
 *(SORT_BY_ALIGNMENT(.noinit))
 *(SORT_BY_ALIGNMENT(.noinit.*))
 .noinit."WEST_TOPDIR/zephyr/subsys/usb/device/usb_work_q.c".0
                0x0000000020009740      0x400 zephyr/libzephyr.a(usb_work_q.c.obj)
                0x0000000020009740                z_usb_work_q_stack
 .noinit."WEST_TOPDIR/zephyr/subsys/shell/backends/shell_uart.c".1
                0x0000000020009b40      0x800 zephyr/libzephyr.a(shell_uart.c.obj)
 .noinit."WEST_TOPDIR/zephyr/subsys/bluetooth/controller/hci/hci_driver.c".1
                0x000000002000a340      0x380 zephyr/subsys/bluetooth/controller/libsubsys__bluetooth__controller.a(hci_driver.c.obj)
 .noinit."WEST_TOPDIR/zephyr/subsys/bluetooth/controller/hci/hci_driver.c".0
                0x000000002000a6c0      0x1c0 zephyr/subsys/bluetooth/controller/libsubsys__bluetooth__controller.a(hci_driver.c.obj)
 .noinit."WEST_TOPDIR/zephyr/subsys/llext/llext_mem.c".kheap_buf_llext_heap
                0x000000002000a880    0x18000 zephyr/subsys/llext/libsubsys__llext.a(llext_mem.c.obj)
                0x000000002000a880                kheap_llext_heap
 .noinit."WEST_TOPDIR/zephyr/drivers/usb/device/usb_dc_nrfx.c".0
                0x0000000020022880      0x400 zephyr/drivers/usb/device/libdrivers__usb__device.a(usb_dc_nrfx.c.obj)
 .noinit."WEST_TOPDIR/zephyr/kernel/init.c".2
                0x0000000020022c80      0x800 zephyr/kernel/libkernel.a(init.c.obj)
                0x0000000020022c80                z_interrupt_stacks
 .noinit."WEST_TOPDIR/zephyr/kernel/init.c".1
                0x0000000020023480      0x140 zephyr/kernel/libkernel.a(init.c.obj)

So it looks like: the address is somewhere within llext: .noinit."WEST_TOPDIR/zephyr/subsys/llext/llext_mem.c".kheap_buf_llext_heap

However, suppose the fault is in the sketch code (either in INO or library). For example if it is in the library readTemerature
The map file shows:

 .text._ZN11HS300xClass15readTemperatureEi
                0x00000000000003ac       0x64 C:\Users\kurte\AppData\Local\arduino\sketches\49F536578E94A5F7D2A96AD76EFA892E\libraries\Arduino_HS300x\HS300x.cpp.o
                0x00000000000003ac                HS300xClass::readTemperature(int)

My main question is: Is there an easy way to map the fault address, back to the location contained within the map file for the sketch.
Maybe something like: subtract the address X from the zephyr map from the fault address and that should give you address...

Thanks

facchinm commented

@pillo79 do you have some instruction we can provide for this kind of debugging?

pillo79 commented

@KurtE yep, unfortunately mapping addresses in sketches is complicated by the fact that the sketch load address can potentially change every boot, so the map file is useless without the actual positions of each section in memory.
As you discovered, the only info you can get is "it's somewhere in the LLEXT heap". Very helpful! 👎

Zephyr very recently accepted a PR that makes the detailed mapping information available in the LLEXT debug log; this can be used to set up gdb and have a proper interactive debugging session. You can read how to use this here, I will merge this functionality in this repo ASAP.

Next steps are to make this information available to sketches (so that they can easily print the same table on Serial), and make a command-line decoder that allows to easily convert those addresses given the compiled sketch file and that map information.

  • 👍1