loader.efi netboot regression in 2023.11
nwf-msr opened this issue · 10 comments
With apologies, I hadn't tested netbooting this release on the MSRC cluster. It looks like we've got a regression. Using the https://github.com/microsoft/msr-morello-automation tooling using HTTP netboot, 2022.12 boots fine, but 2023.11 (built locally) dies at the very end of loader.efi with the below wad of complaints. I have tested on both the 1.5 and 1.7 releases of Morello firmware, with no apparent difference. Fortunately, loader_lua.efi from 2022.12 continues to function, so that's pretty convenient as workarounds go.
/boot/kernel/geom_eli.ko text=0x57f6 text=0xac78 data=0x1050+0x170 TcpOnAppAbort: connection reset issued by application for TCB FC2D3E98
0x8+0x2148TcpOnAppAbort: connection reset issued by application for TCB FC2D3E98
TcpInput: Discard a packet
TcpInput: Discard a packet
TcpInput: Discard a packet
+0x
8+0x1656
TcpOnAppAbort: connection reset issued by application for TCB FC2D3E98
Using DTB provided by EFI at 0xfd7a6000.
ConvertPages: Incompatible memory types, the pages to allocate have been allocated
Synchronous Exception at 0x00000000F327C740
Synchronous Exception at 0x00000000F327C740
PC 0x0000F327C740
PC 0x0000F7D310DC
PC 0x0000F7D3A498
PC 0x0000F7D3ACE8
PC 0x0000F7D709D0
PC 0x0000F7D4D374
PC 0x0000F7D6555C
PC 0x0000F7D4D5CC
PC 0x0000F7D4C56C
PC 0x0000F7D4DB54
PC 0x0000F7D499C8
PC 0x0000F7D46770
PC 0x0000F7D3C76C
PC 0x0000F7D707C8
PC 0x0000F7D4D0A0
PC 0x0000F7D673A8
PC 0x0000F7D4D5CC
PC 0x0000F7D4C56C
PC 0x0000F7D4DB54
PC 0x0000F7D499C8
PC 0x0000F7D468A0
PC 0x0000F7D3C4C0
PC 0x0000F7D34124
PC 0x0000F7D320A4
PC 0x0000F7D3A550
PC 0x0000F7E32044 (0x0000F7E1D000+0x00015044) [ 1] UiApp.dll
PC 0x0000F7E38BD0 (0x0000F7E1D000+0x0001BBD0) [ 1] UiApp.dll
PC 0x0000FD545A10 (0x0000FD52D000+0x00018A10) [ 2] SetupBrowser.dll
PC 0x0000FD548D98 (0x0000FD52D000+0x0001BD98) [ 2] SetupBrowser.dll
PC 0x0000FD5337B4 (0x0000FD52D000+0x000067B4) [ 2] SetupBrowser.dll
PC 0x0000F7E2CEF4 (0x0000F7E1D000+0x0000FEF4) [ 3] UiApp.dll
PC 0x0000F7E2CDC4 (0x0000F7E1D000+0x0000FDC4) [ 3] UiApp.dll
PC 0x0000F7E1ED4C (0x0000F7E1D000+0x00001D4C) [ 3] UiApp.dll
PC 0x0000F7E1E5F4 (0x0000F7E1D000+0x000015F4) [ 3] UiApp.dll
PC 0x0000FD76E2CC (0x0000FD74F000+0x0001F2CC) [ 4] DxeCore.dll
PC 0x0000FD51C074 (0x0000FD50D000+0x0000F074) [ 5] BdsDxe.dll
PC 0x0000FD51FCC4 (0x0000FD50D000+0x00012CC4) [ 5] BdsDxe.dll
PC 0x0000FD516EAC (0x0000FD50D000+0x00009EAC) [ 5] BdsDxe.dll
PC 0x0000FD75AE60 (0x0000FD74F000+0x0000BE60) [ 6] DxeCore.dll
PC 0x0000FD751B80 (0x0000FD74F000+0x00002B80) [ 6] DxeCore.dll
PC 0x0000FD75181C (0x0000FD74F000+0x0000281C) [ 6] DxeCore.dll
[ 1] /builds/morello/manifest/bsp/uefi/edk2/Build/morellosoc/DEBUG_CLANGDWARF/AARCH64/MdeModulePkg/Application/UiApp/UiApp/DEBUG/UiApp.dll
[ 2] /builds/morello/manifest/bsp/uefi/edk2/Build/morellosoc/DEBUG_CLANGDWARF/AARCH64/MdeModulePkg/Universal/SetupBrowserDxe/SetupBrowserDxe/DEBUG/SetupBrowser.dll
[ 3] /builds/morello/manifest/bsp/uefi/edk2/Build/morellosoc/DEBUG_CLANGDWARF/AARCH64/MdeModulePkg/Application/UiApp/UiApp/DEBUG/UiApp.dll
[ 4] /builds/morello/manifest/bsp/uefi/edk2/Build/morellosoc/DEBUG_CLANGDWARF/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
[ 5] /builds/morello/manifest/bsp/uefi/edk2/Build/morellosoc/DEBUG_CLANGDWARF/AARCH64/MdeModulePkg/Universal/BdsDxe/BdsDxe/DEBUG/BdsDxe.dll
[ 6] /builds/morello/manifest/bsp/uefi/edk2/Build/morellosoc/DEBUG_CLANGDWARF/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
X0 0x00000000F7DF0C80 X1 0x00000000F327C740 X2 0x0000000000000003 X3 0x00000000F7DF0CB0
X4 0x0000000000000000 X5 0x0000000000000041 X6 0x00000000F7D875E0 X7 0xFF9F000000000B3F
X8 0x00000000FFFFFFFB X9 0x0000000000000075 X10 0x00000000F7DE09F8 X11 0xFEFEFEFEFEFEFEFF
X12 0x0000000000000008 X13 0x0000000000000000 X14 0x00000000F327C490 X15 0x0000000000000030
X16 0x0000000000000000 X17 0x0000000000000020 X18 0x0000000000000000 X19 0x00000000F22012B0
X20 0x0000000000001000 X21 0x0000000000000002 X22 0x00000000F7DF0D18 X23 0x00000000F7DF0D38
X24 0x00000000F32796B0 X25 0x0000000000000001 X26 0x00000000F3278AA0 X27 0x00000000010000C4
X28 0x00000000F3278A80 FP 0x00000000F7DF0CA0 LR 0x00000000F7D310DC
V0 0xAFAFAFAFAFAFAFAF AFAFAFAFAFAFAFAF V1 0x00000000F7DF06A0 00000000F7DF0720
V2 0x0000000000000000 0000000000000000 V3 0x0000000000000000 0000000000000000
V4 0x0000000000000000 0000000000000000 V5 0x0000000000000000 0000000000000000
V6 0x0000000000000000 0000000000000000 V7 0x0000000000000000 0000000000000000
V8 0x0000000000000000 0000000000000000 V9 0x0000000000000000 0000000000000000
V10 0x0000000000000000 0000000000000000 V11 0x0000000000000000 0000000000000000
V12 0x0000000000000000 0000000000000000 V13 0x0000000000000000 0000000000000000
V14 0x0000000000000000 0000000000000000 V15 0x0000000000000000 0000000000000000
V16 0x0000000000000000 0000000000000000 V17 0x0000000000000000 0000000000000000
V18 0x0000000000000000 0000000000000000 V19 0x0000000000000000 0000000000000000
V20 0x0000000000000000 0000000000000000 V21 0x0000000000000000 0000000000000000
V22 0x0000000000000000 0000000000000000 V23 0x0000000000000000 0000000000000000
V24 0x0000000000000000 0000000000000000 V25 0x0000000000000000 0000000000000000
V26 0x0000000000000000 0000000000000000 V27 0x0000000000000000 0000000000000000
V28 0x0000000000000000 0000000000000000 V29 0x0000000000000000 0000000000000000
V30 0x0000000000000000 0000000000000000 V31 0x0000000000000000 0000000000000000
SP 0x00000000F7DF0C00 ELR 0x00000000F327C740 SPSR 0x80000209 FPSR 0x00000000
ESR 0x02000000 FAR 0x90A84966BC426F81
ESR : EC 0x00 IL 0x1 ISS 0x00000000
Stack dump:
00000F7DF0B00: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
00000F7DF0B20: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
00000F7DF0B40: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
00000F7DF0B60: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
00000F7DF0B80: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
00000F7DF0BA0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
00000F7DF0BC0: 0000000000000000 0000000000000000 00000000FD596A9C 0000000000000308
00000F7DF0BE0: 0000000000000000 0000000002000000 90A84966BC426F81 00000000F22012B0
> 00000F7DF0C00: 0000000000000000 0000000000000000 00000000F7DF0CE0 00000000F7D88404
00000F7DF0C20: 00000000F3278A80 00001000010000C4 00000000F3278AA0 00000000F3278AA8
00000F7DF0C40: 00000000F7DF0C80 00000000F7D8867C 00000000F327C450 0000000000000000
00000F7DF0C60: 00000000F3229B80 0000000000000003 0000000000000002 00000000F7DF0D00
00000F7DF0C80: 00000000F7DF0CB0 00000000F7D86700 00000000F327C400 0000000000000000
00000F7DF0CA0: 00000000F7DF0D20 00000000F7D3A498 00000000F3278A80 00000000010000C4
00000F7DF0CC0: 00000000F3278AA0 00000000F3278AA8 00000000F32796B0 00000000F7D7098C
00000F7DF0CE0: 00000000F3278AE0 0000000000000002 00000000EC600800 00000000F327C740
ASSERT [ArmCpuDxe] DefaultExceptionHandler.c(333): ((BOOLEAN)(0==1))
If it's dying in loader that's an upstream bug, we don't turn anything CHERI on there. Can you please verify it's reproducible with 14.0-RELEASE and/or recent 15-CURRENT?
Yes, this is an upstream regression. Bisection (full sequence of tests below) points at 75b7d39e ("stand: efi_fmtdev can be reduced to devformat") being the culprit, though at a glance it's hard to see why.
git bisect bad 525ecfdad597980ea4cd59238e24c8530dbcd31d
git bisect good fc952ac2212b121aa6eefc273f5960ec3e0a466d
git bisect good 8824cbace389c440394bb9ea6c127d0f8f85538b
git bisect good 3fc5f6b0abf5d5e57a7f171a87b1fb7de63b096f
git bisect good 1c915de99108487f58f338ddf1287bda489cb73f
git bisect bad efbf827ae461a66402acb191c036623ce1ee608f
git bisect good 16ccda32623a98cf4a2ab3804b5ae6277c01395e
git bisect bad 03c44354fc9e82f89d6e323d5f20ab953854812d
git bisect good 09a31487c1bdcb5dc8780fdd7e78540d686a3854
git bisect good 9b2453430a8dc49d34be7560ad71ff568b973672
git bisect good 5c505b9d8166d997015d3e841846b5222375a1e7
git bisect good ec6d9023e56aebe0c3fe8fb7d15512097a20df8a
git bisect bad 26e5e3ecbd61d82b369a0ce6a83297e95526a691
git bisect good e9db6f25b5a6bb9ca76351c88720cca2ffcb6604
git bisect bad e36bd9b6cffb13f2ff50bd3551ff20188d5b8099
git bisect bad 75b7d39e116f79563ea3d8d18f9bf3141f89a712
It's a bit odd that you've hit the MFC commit to stable/13 rather than the one to main...
I will say it's not clear those functions are equivalent except in the simplest cases and the commit doesn't explain why only those cases can happen.
devformat should produce exactly the same results. If not, it's a bug in the dev->d_dev formatting routine (which should default to the default: case for http booting. devformat uses devsw->dv_fmtdev if it exists, and defaults to
snprintf(name, sizeof(name), "%s%d:", d->d_dev->dv_name, d->d_unit);
if not, which is the same as the default case which was removed i 75b7d3...
- default:
- sprintf(buf, "%s%d:", dev->d_dev->dv_name, dev->d_unit);
- break;
I think only DEVT_NONE type devices are different.
Since only a few devices in the devsw have fmtdev, they should be teh same. disk has a different one, and zfs has a different one as well....
And I know others have network booted.
One could revert that one change, and then unrevert the replaced calls to efi_fmtdev one at a time (there's only 3) to see which one goes south, and what the devdesc that's passed into devformat() function looks like.
Also, it would be nice to get a symbolic traceback on what's happening. I have some libtraceback code written, but it doesn't quite work so I've not pushed it into Upstream FreeBSD... but knowing which of these calls to efi_fmtdev dies might suffice.
Manually symbolizing...
Synchronous Exception at 0x00000000F327C740
PC 0x0000F327C740
PC 0x0000F7D310DC bi_load
PC 0x0000F7D3A498 elf64_exec
PC 0x0000F7D3ACE8 command_boot
PC 0x0000F7D709D0 lua_perform
PC 0x0000F7D4D374 luaD_precall
PC 0x0000F7D6555C luaV_execute
PC 0x0000F7D4D5CC luaD_callnoyield
PC 0x0000F7D4C56C luaD_rawrunprotected
PC 0x0000F7D4DB54 luaD_pcall
PC 0x0000F7D499C8 lua_pcallk
PC 0x0000F7D46770 interp_include
PC 0x0000F7D3C76C command_include
PC 0x0000F7D707C8 lua_command
PC 0x0000F7D4D0A0 luaD_pretailcall
PC 0x0000F7D673A8 luaV_execute
PC 0x0000F7D4D5CC luaD_callnoyield
PC 0x0000F7D4C56C luaD_rawrunprotected
PC 0x0000F7D4DB54 luaD_pcall
PC 0x0000F7D499C8 lua_pcall
PC 0x0000F7D468A0 interp_run
PC 0x0000F7D3C4C0 interact
PC 0x0000F7D34124 main
PC 0x0000F7D320A4 efi_main
PC 0x0000F7D3A550 _start
In particular,
; getrootmount(devformat(rootdev));
10d8: 3c 7c 01 94 bl 0x601c8 <devformat>
10dc: 06 26 00 94 bl 0xa8f4 <getrootmount>
but 0x0000F327C740 looks like it's somewhere else in EFI and so the trail runs pretty cold, though I think it's safe to say that we hit the d->d_dev->dv_fmtdev != NULL case. Oddly, looking at the rest of the EFI debug messages, I don't see anything getting loaded that far south.
So the exception is at 0x0000F327C740, which is well outside the range of all the other addresses on the stack. But it has survived the call to devformat...
I'm not sure what the device should be... but one test would be to set vfs.root.mountfrom to what you think it should be before issuing 'boot'. This would let us know if it was the return value (and/or the function itself) that's causing this or something else. Worst case, if you set this to something non-sensical, the kernel will just not be able to fine /, so its safe to test to see if the return value from devformat() changes in a way that runs of off the cliff we hit.
I'm not familar with cheri enough to know, but are we running in capabilities which trap on buffer overflows, or are we still in mixed mode where they might be possible?
loader is built as a plain AArch64 binary, no capabilities allowed, since the firmware doesn't save/restore them currently on traps so they would be clobbered at arbitrary points. We only enable capability use in the kernel's locore.
OK, so we can't rule out an overflow or something similar....
So the exception is at 0x0000F327C740, which is well outside the range of all the other addresses on the stack. But it has survived the call to devformat...
I don't think that's right; the return address is 0x0000F7D310DC, which means that bi_load is still "in" its call to devformat, which has (possibly transitively) tail-called to 0x0000F327C740.