limine-bootloader/limine

Potentially broken EFI memory map

phip1611 opened this issue · 15 comments

phip1611 commented

Hi, I'm currently not sure where the problem is and maybe you have something helpful to say. I think Limine might have a bug when it comes to the EFI memory map and Multiboot2.

VM configuration

I'm booting a QEMU x86_64 VM with the q35 machine type, 128MB of RAM, and OVMF as UEFI-firmware in version 202402. The firmware boots Limine 7.5.2 from a hybrid ISO image. (I think that Limine boots the legacy way as OVMF has the Compatibility Support Module (CSM) active, hence, the .EFI file is not touched.)

Limine Payload

Limine than boots a Multiboot2 binary which is mapped straight into memory without any relocation at 16MB. The handoff happens in i386 32-bit protected machine state without paging.

In that payload, the EFI memory map coming from the Multiboot Boot Information (MBI) is corrupt. The relevant output, containing the regular memory map (which seems fine) and the EFI memory map, is the following:

The tags from the MBI:

Note that the efi_memory_map seems to be suspiously big.

[DEBUG: multiboot2_payload/src/main.rs@32]: multiboot2_hdr=0x00010000, multiboot2_magic=0x36d76289
Multiboot2BootInformation {
    start_address: 0x10000,
    end_address: 0x11a70,
    total_size: 0x1a70,
    // ...
    efi_memory_map: Some(
        EFIMemoryMapTag {
            typ: EfiMmap,
            size: 0x1540,
            desc_size: 0x30,
            desc_version: 0x1,
            memory_map: "<data...>",
        },
    ),
    // ...
    memory_map: Some(
        MemoryMapTag {
            typ: Mmap,
            size: 0x268,
            entry_size: 0x18,
            entry_version: 0x0,
            areas: [
                // ...
            ],
        },
    ),
}

Printed legacy memory map (seems fine):

[ INFO: multiboot2_payload/src/verify/mod.rs@20]: loaded by Limine
Memory Map (legacy):
  0x0000000000 - 0x00000a0000 (  0.6 MiB Available)
  0x0000100000 - 0x0000800000 (  7.0 MiB Available)
  0x0000800000 - 0x0000808000 (  0.0 MiB ReservedHibernate)
  0x0000808000 - 0x000080b000 (  0.0 MiB Available)
  0x000080b000 - 0x000080c000 (  0.0 MiB ReservedHibernate)
  0x000080c000 - 0x0000810000 (  0.0 MiB Available)
  0x0000810000 - 0x0000900000 (  0.9 MiB ReservedHibernate)
  0x0000900000 - 0x0006448000 ( 91.3 MiB Available)
  0x0006448000 - 0x000644d000 (  0.0 MiB Reserved)
  0x000644d000 - 0x0006456000 (  0.0 MiB Available)
  0x0006456000 - 0x000648e000 (  0.2 MiB Reserved)
  0x000648e000 - 0x00064bc000 (  0.2 MiB Available)
  0x00064bc000 - 0x00064c7000 (  0.0 MiB Reserved)
  0x00064c7000 - 0x00074ed000 ( 16.1 MiB Available)
  0x00074ed000 - 0x00075ed000 (  1.0 MiB Reserved)
  0x00075ed000 - 0x00076ed000 (  1.0 MiB Reserved)
  0x00076ed000 - 0x000776d000 (  0.5 MiB Reserved)
  0x000776d000 - 0x000777f000 (  0.1 MiB AcpiAvailable)
  0x000777f000 - 0x00077ff000 (  0.5 MiB ReservedHibernate)
  0x00077ff000 - 0x0007ef4000 (  7.0 MiB Available)
  0x0007ef4000 - 0x0007f78000 (  0.5 MiB Reserved)
  0x0007f78000 - 0x0008000000 (  0.5 MiB ReservedHibernate)
  0x00e0000000 - 0x00f0000000 (256.0 MiB Reserved)
  0x00feffc000 - 0x00ff000000 (  0.0 MiB Reserved)
  0xfd00000000 - 0x10000000000 (12288.0 MiB Reserved)

Printed EFI memory map (seems odd):

Memory Map (EFI):
  0x0000000000000000 - 0x0000100000000000 (16777216.0 MiB BOOT_SERVICES_CODE)
  0x0000100000000000 - 0x000a000000000000 (2667577344.0 MiB CONVENTIONAL)
  0x0010000000000000 - 0x0080000000000000 (30064771072.0 MiB CONVENTIONAL)
  0x0080000000000000 - 0x0080800000000000 (134217728.0 MiB ACPI_NON_VOLATILE)
  0x0080800000000000 - 0x0080b00000000000 (50331648.0 MiB CONVENTIONAL)
  0x0080b00000000000 - 0x0080c00000000000 (16777216.0 MiB ACPI_NON_VOLATILE)
  0x0080c00000000000 - 0x0081000000000000 (67108864.0 MiB CONVENTIONAL)
  0x0081000000000000 - 0x0090000000000000 (4026531840.0 MiB ACPI_NON_VOLATILE)
  0x0090000000000000 - 0x0178000000000000 (62277025792.0 MiB BOOT_SERVICES_DATA)
  0x0178000000000000 - 0x0237500000000000 (51355058176.0 MiB CONVENTIONAL)
  0x0237500000000000 - 0x03b7500000000000 (103079215104.0 MiB CONVENTIONAL)
  0x03b7500000000000 - 0x03b9500000000000 (536870912.0 MiB BOOT_SERVICES_DATA)
  0x03b9500000000000 - 0x03c5600000000000 (3238002688.0 MiB CONVENTIONAL)
  0x03c5600000000000 - 0x0644800000000000 (171563810816.0 MiB CONVENTIONAL)
  0x0644800000000000 - 0x0644d00000000000 (83886080.0 MiB LOADER_DATA)
  0x0644d00000000000 - 0x0645600000000000 (150994944.0 MiB BOOT_SERVICES_DATA)
  0x0645600000000000 - 0x0648e00000000000 (939524096.0 MiB LOADER_CODE)
  0x0648e00000000000 - 0x064c700000000000 (956301312.0 MiB LOADER_DATA)
  0x064c700000000000 - 0x068ea00000000000 (17767071744.0 MiB BOOT_SERVICES_DATA)
  0x068ea00000000000 - 0x0699200000000000 (2818572288.0 MiB BOOT_SERVICES_CODE)
  0x0699200000000000 - 0x06a0600000000000 (1946157056.0 MiB BOOT_SERVICES_DATA)
  0x06a0600000000000 - 0x06a1f00000000000 (419430400.0 MiB BOOT_SERVICES_CODE)
  0x06a1f00000000000 - 0x06a2400000000000 (83886080.0 MiB BOOT_SERVICES_DATA)
  0x06a2400000000000 - 0x06a3400000000000 (268435456.0 MiB BOOT_SERVICES_CODE)
  0x06a3400000000000 - 0x06a3800000000000 (67108864.0 MiB BOOT_SERVICES_DATA)
  0x06a3800000000000 - 0x06a5100000000000 (419430400.0 MiB BOOT_SERVICES_CODE)
  0x06a5100000000000 - 0x06a5200000000000 (16777216.0 MiB BOOT_SERVICES_DATA)
  0x06a5200000000000 - 0x06a5500000000000 (50331648.0 MiB BOOT_SERVICES_CODE)
  0x06a5500000000000 - 0x06a5600000000000 (16777216.0 MiB BOOT_SERVICES_DATA)
  0x06a5600000000000 - 0x06a8900000000000 (855638016.0 MiB BOOT_SERVICES_CODE)
  0x06a8900000000000 - 0x06a9300000000000 (167772160.0 MiB BOOT_SERVICES_DATA)
  0x06a9300000000000 - 0x06a9900000000000 (100663296.0 MiB BOOT_SERVICES_CODE)
  0x06a9900000000000 - 0x06a9f00000000000 (100663296.0 MiB BOOT_SERVICES_DATA)
  0x06a9f00000000000 - 0x06aaf00000000000 (268435456.0 MiB BOOT_SERVICES_CODE)
  0x06aaf00000000000 - 0x06ab200000000000 (50331648.0 MiB BOOT_SERVICES_DATA)
  0x06ab200000000000 - 0x06acd00000000000 (452984832.0 MiB BOOT_SERVICES_CODE)
  0x06acd00000000000 - 0x06ace00000000000 (16777216.0 MiB BOOT_SERVICES_DATA)
  0x06ace00000000000 - 0x06b0000000000000 (838860800.0 MiB BOOT_SERVICES_CODE)
  0x06b0000000000000 - 0x06b0300000000000 (50331648.0 MiB BOOT_SERVICES_DATA)
  0x06b0300000000000 - 0x06b0a00000000000 (117440512.0 MiB BOOT_SERVICES_CODE)
  0x06b0a00000000000 - 0x06b0d00000000000 (50331648.0 MiB BOOT_SERVICES_DATA)
  0x06b0d00000000000 - 0x06b1c00000000000 (251658240.0 MiB BOOT_SERVICES_CODE)
  0x06b1c00000000000 - 0x06b1d00000000000 (16777216.0 MiB BOOT_SERVICES_DATA)
  0x06b1d00000000000 - 0x06b1f00000000000 (33554432.0 MiB BOOT_SERVICES_CODE)
  0x06b1f00000000000 - 0x06b2100000000000 (33554432.0 MiB BOOT_SERVICES_DATA)
  0x06b2100000000000 - 0x06b2700000000000 (100663296.0 MiB BOOT_SERVICES_CODE)
  0x06b2700000000000 - 0x06b2a00000000000 (50331648.0 MiB BOOT_SERVICES_DATA)
  0x06b2a00000000000 - 0x06b2f00000000000 (83886080.0 MiB BOOT_SERVICES_CODE)
  0x06b2f00000000000 - 0x06b3100000000000 (33554432.0 MiB BOOT_SERVICES_DATA)
  0x06b3100000000000 - 0x06b3d00000000000 (201326592.0 MiB BOOT_SERVICES_CODE)
  0x06b3d00000000000 - 0x06b3f00000000000 (33554432.0 MiB BOOT_SERVICES_DATA)
  0x06b3f00000000000 - 0x06b4d00000000000 (234881024.0 MiB BOOT_SERVICES_CODE)
  0x06b4d00000000000 - 0x06b4f00000000000 (33554432.0 MiB BOOT_SERVICES_DATA)
  0x06b4f00000000000 - 0x06b5100000000000 (33554432.0 MiB BOOT_SERVICES_CODE)
  0x06b5100000000000 - 0x06b5300000000000 (33554432.0 MiB BOOT_SERVICES_DATA)
  0x06b5300000000000 - 0x06b6f00000000000 (469762048.0 MiB BOOT_SERVICES_CODE)
  0x06b6f00000000000 - 0x06b7200000000000 (50331648.0 MiB BOOT_SERVICES_DATA)
  0x06b7200000000000 - 0x06b8700000000000 (352321536.0 MiB BOOT_SERVICES_CODE)
  0x06b8700000000000 - 0x06b8a00000000000 (50331648.0 MiB BOOT_SERVICES_DATA)
  0x06b8a00000000000 - 0x06b9100000000000 (117440512.0 MiB BOOT_SERVICES_CODE)
  0x06b9100000000000 - 0x06b9900000000000 (134217728.0 MiB BOOT_SERVICES_DATA)
  0x06b9900000000000 - 0x06b9d00000000000 (67108864.0 MiB BOOT_SERVICES_CODE)
  0x06b9d00000000000 - 0x06ba300000000000 (100663296.0 MiB BOOT_SERVICES_DATA)
  0x06ba300000000000 - 0x06ba700000000000 (67108864.0 MiB BOOT_SERVICES_CODE)
  0x06ba700000000000 - 0x06baa00000000000 (50331648.0 MiB BOOT_SERVICES_DATA)
  0x06baa00000000000 - 0x06be900000000000 (1056964608.0 MiB BOOT_SERVICES_CODE)
  0x06be900000000000 - 0x06bf000000000000 (117440512.0 MiB BOOT_SERVICES_DATA)
  0x06bf000000000000 - 0x06bf300000000000 (50331648.0 MiB BOOT_SERVICES_CODE)
  0x06bf300000000000 - 0x06bf600000000000 (50331648.0 MiB BOOT_SERVICES_DATA)
  0x06bf600000000000 - 0x06bfb00000000000 (83886080.0 MiB BOOT_SERVICES_CODE)
  0x06bfb00000000000 - 0x06bfd00000000000 (33554432.0 MiB BOOT_SERVICES_DATA)
  0x06bfd00000000000 - 0x06c0000000000000 (50331648.0 MiB BOOT_SERVICES_CODE)
  0x06c0000000000000 - 0x06e0300000000000 (8640266240.0 MiB BOOT_SERVICES_DATA)
  0x06e0300000000000 - 0x06e1800000000000 (352321536.0 MiB BOOT_SERVICES_CODE)
  0x06e1800000000000 - 0x06e1900000000000 (16777216.0 MiB BOOT_SERVICES_DATA)
  0x06e1900000000000 - 0x06e1d00000000000 (67108864.0 MiB BOOT_SERVICES_CODE)
  0x06e1d00000000000 - 0x06e1f00000000000 (33554432.0 MiB BOOT_SERVICES_DATA)
  0x06e1f00000000000 - 0x06e2700000000000 (134217728.0 MiB BOOT_SERVICES_CODE)
  0x06e2700000000000 - 0x06e2a00000000000 (50331648.0 MiB BOOT_SERVICES_DATA)
  0x06e2a00000000000 - 0x06e2b00000000000 (16777216.0 MiB BOOT_SERVICES_CODE)
  0x06e2b00000000000 - 0x06e2d00000000000 (33554432.0 MiB BOOT_SERVICES_DATA)
  0x06e2d00000000000 - 0x06e3100000000000 (67108864.0 MiB BOOT_SERVICES_CODE)
  0x06e3100000000000 - 0x06e3300000000000 (33554432.0 MiB BOOT_SERVICES_DATA)
  0x06e3300000000000 - 0x06e4a00000000000 (385875968.0 MiB BOOT_SERVICES_CODE)
  0x06e4a00000000000 - 0x06e4b00000000000 (16777216.0 MiB BOOT_SERVICES_DATA)
  0x06e4b00000000000 - 0x06e4c00000000000 (16777216.0 MiB BOOT_SERVICES_CODE)
  0x06e4c00000000000 - 0x0726000000000000 (17515413504.0 MiB BOOT_SERVICES_DATA)
  0x0726000000000000 - 0x0726600000000000 (100663296.0 MiB BOOT_SERVICES_CODE)
  0x0726600000000000 - 0x0726900000000000 (50331648.0 MiB BOOT_SERVICES_DATA)
  0x0726900000000000 - 0x0726a00000000000 (16777216.0 MiB BOOT_SERVICES_CODE)
  0x0726a00000000000 - 0x0726b00000000000 (16777216.0 MiB BOOT_SERVICES_DATA)
  0x0726b00000000000 - 0x0727500000000000 (167772160.0 MiB BOOT_SERVICES_CODE)
  0x0727500000000000 - 0x0727700000000000 (33554432.0 MiB BOOT_SERVICES_DATA)
  0x0727700000000000 - 0x0727900000000000 (33554432.0 MiB BOOT_SERVICES_CODE)
  0x0727900000000000 - 0x0727a00000000000 (16777216.0 MiB BOOT_SERVICES_DATA)
  0x0727a00000000000 - 0x0727d00000000000 (50331648.0 MiB BOOT_SERVICES_CODE)
  0x0727d00000000000 - 0x074ed00000000000 (10468982784.0 MiB BOOT_SERVICES_DATA)
  0x074ed00000000000 - 0x075ed00000000000 (4294967296.0 MiB RUNTIME_SERVICES_DATA)
  0x075ed00000000000 - 0x076ed00000000000 (4294967296.0 MiB RUNTIME_SERVICES_CODE)
  0x076ed00000000000 - 0x0776d00000000000 (2147483648.0 MiB RESERVED)
  0x0776d00000000000 - 0x0777f00000000000 (301989888.0 MiB ACPI_RECLAIM)
  0x0777f00000000000 - 0x077ff00000000000 (2147483648.0 MiB ACPI_NON_VOLATILE)
  0x077ff00000000000 - 0x07e0000000000000 (25786580992.0 MiB BOOT_SERVICES_DATA)
  0x07e0000000000000 - 0x07e8700000000000 (2264924160.0 MiB CONVENTIONAL)
  0x07e8700000000000 - 0x07ea700000000000 (536870912.0 MiB BOOT_SERVICES_DATA)
  0x07ea700000000000 - 0x07eca00000000000 (587202560.0 MiB BOOT_SERVICES_CODE)
  0x07eca00000000000 - 0x07edb00000000000 (285212672.0 MiB BOOT_SERVICES_DATA)
  0x07edb00000000000 - 0x07ef400000000000 (419430400.0 MiB BOOT_SERVICES_CODE)
  0x07ef400000000000 - 0x07f7800000000000 (2214592512.0 MiB RUNTIME_SERVICES_DATA)
  0x07f7800000000000 - 0x0800000000000000 (2281701376.0 MiB ACPI_NON_VOLATILE)
  0xe000000000000000 - 0xf000000000000000 (1099511627776.0 MiB RESERVED)
  0xfeffc00000000000 - 0xff00000000000000 (67108864.0 MiB RESERVED)
  0x0000000000000000 - 0x0000000000000000 (  0.0 MiB RESERVED)

I'm confident that it is not a bug of my code. Especially the multiboot2 code is heavily unit- and integration tested. And I don't have any other weird side-effects.

Do you think, it could be that Limine somehow messes with the EFI mmap?

phip1611 commented

I run Limine with VERBOSE=yes, but all it tells me is:

multiboot2: Loading kernel `boot:///kernel`...
acpi: Found SMBIOS 32-bit entry point at 0x753f000
acpi: Found SMBIOS 64-bit entry point at 0x753d000

According to the boot information, the boot services are already excited (the boot services not exited tag is not present), an efi_std64 tag is present, and an efi_ih32 tag (bugfix PR open) is present.

mintsuki commented

Hi, I just tested this, it seems fine to me, booting a test kernel written in C.

phip1611 commented

Okay, thanks!

I'll do more experiments, including a hand-off without exited boot services, more logging in Limine etc, and come back to you. Likely within the next 4 days.

mintsuki commented

Alright, mind you though that Limine does not support hand-off without exited boot services (sort of by design, because it is a useless, rarely used mode of multiboot2).

phip1611 commented

Ah, oh really? GRUB supports it as well. However, GRUB doesn't support some Multiboot-scenarios that Limine supports on the other hand. 🤷🏻

mintsuki commented

GRUB2 is supposed to be the reference implementation... (let's ignore the places where IT IS the spec and the spec is useless)

phip1611 commented

From my testing, it is hard to get output from Limine. Perhaps I'm missing something. I use the serial output and capture everything in a file with QEMU.

My observations:

  • as soon as term_notready(); in multiboot2.c is called, all following print() are ignored
  • in the serial.txt file, there are never more than 23 lines. This is very odd
  • the serial.txt file is full of ANSI escape sequences - perhaps the problem is rooted there somehow

TL;DR: Getting output is hard. And tips?

--

By using the following patch to print the EFI memory map that Limine sees:

  • moved the block for "Create EFI memory map tag" above "Create framebuffer tag" to be able to use print() (see above)
  • printing the mmap using print() from within limine
diff --git a/common/protos/multiboot2.c b/common/protos/multiboot2.c
index e5f085f6..2e7d13a6 100644
--- a/common/protos/multiboot2.c
+++ b/common/protos/multiboot2.c
@@ -558,6 +558,49 @@ reloc_fail:
         append_tag(info_idx, module_tag);
     }
 
+    //////////////////////////////////////////////
+    // Create EFI memory map tag
+    //////////////////////////////////////////////
+#if defined (UEFI)
+    {
+        if ((efi_mmap_size / efi_desc_size) > MEMMAP_MAX) {
+            panic(false, "multiboot2: too many EFI memory map entries");
+        }
+        if (efi_mmap_size % efi_desc_size != 0) {
+            panic(false, "multiboot2: invalid EFI memory map size");
+        }
+
+        // Create the EFI memory map tag.
+        uint32_t size = sizeof(struct multiboot_tag_efi_mmap) + efi_mmap_size;
+        struct multiboot_tag_efi_mmap *mmap_tag = (struct multiboot_tag_efi_mmap *)(mb2_info + info_idx);
+
+        mmap_tag->type = MULTIBOOT_TAG_TYPE_EFI_MMAP;
+        mmap_tag->descr_vers = efi_desc_ver;
+        mmap_tag->descr_size = efi_desc_size;
+        mmap_tag->size = size;
+
+        print("multiboot2: efi_mmap: \n");
+
+        // Copy over the EFI memory map.
+        memcpy(mmap_tag->efi_mmap, efi_mmap, efi_mmap_size);
+        append_tag(info_idx, mmap_tag);
+
+        print("UEFI memory map: \n");
+        print("  <#> <phys> <size> <type> <attr>\n");
+        int n_entries = efi_mmap_size / efi_desc_size;
+        print("number of entries   = %d\n", n_entries);
+        print("tag size            = %x\n", size);
+        print("efi memory map size = %x\n", efi_mmap_size);
+
+        // Limine serial log output is weird. In the serial.txt file, there
+        // are never printed more than
+        for (int i = 0; i < n_entries; i++) {
+            EFI_MEMORY_DESCRIPTOR * e = ((char *) efi_mmap) + i * efi_desc_size;
+            print("  [%d] %x %x %x %x: \n", i, e->PhysicalStart, e->NumberOfPages * 4096, e->Type, e->Attribute);
+        }
+    }
+#endif
+
     //////////////////////////////////////////////
     // Create command line tag
     //////////////////////////////////////////////
@@ -857,29 +900,7 @@ skip_modeset:;
         append_tag(info_idx, tag);
     }
 
-    //////////////////////////////////////////////
-    // Create EFI memory map tag
-    //////////////////////////////////////////////
-#if defined (UEFI)
-    {
-        if ((efi_mmap_size / efi_desc_size) > MEMMAP_MAX) {
-            panic(false, "multiboot2: too many EFI memory map entries");
-        }
 
-        // Create the EFI memory map tag.
-        uint32_t size = sizeof(struct multiboot_tag_efi_mmap) + efi_mmap_size;
-        struct multiboot_tag_efi_mmap *mmap_tag = (struct multiboot_tag_efi_mmap *)(mb2_info + info_idx);
-
-        mmap_tag->type = MULTIBOOT_TAG_TYPE_EFI_MMAP;
-        mmap_tag->descr_vers = efi_desc_ver;
-        mmap_tag->descr_size = efi_desc_size;
-        mmap_tag->size = size;
-
-        // Copy over the EFI memory map.
-        memcpy(mmap_tag->efi_mmap, efi_mmap, efi_mmap_size);
-        append_tag(info_idx, mmap_tag);
-    }
-#endif
 
     //////////////////////////////////////////////
     // Create network info tag

I could verify that at least the first 8 entries (couldn't get more output, see above) are equal to what my Rust binary kernel finds in multiboot_tag_mmap (not multiboot_tag_efi_mmap).

However, multiboot_tag_efi_mmap reports the correct length and desc_size, but it contains just gibberish.

At this point I can say that it is not the multiboot2 crate as it is thoroughly tested.

Observations:

  1. I'm surprised that Limine provides multiboot_tag_mmap by transforming the EFI memory map into the old format
  2. I think Limine somehow messes with the EFI memory.

I wish I could provide you a minimal example. For now, all I got is this:

which you can run using nix-shell --run "integration-test/run.sh"; cat integration-test/serial.txt

In the output (if you scroll a little up) you can see how the corrupted EFI memory map is printed).

mintsuki commented

I added prints here

limine/test/multiboot2.c

Line 102 in c204af4

case MULTIBOOT_TAG_TYPE_EFI_MMAP: {
and it outputs perfectly fine.

mintsuki commented

Just make the test image and run the multiboot2 test from the boot menu, you'll get console output via port 0xe9.

phip1611 commented

Ah, the debugcon device. Nice, thanks! Didn't know the function e9_printf() exists.

phip1611 commented

PS: I was talking about output from the bootloader, not the multiboot2 test binary! I had to move e9print.h and e9print.c to common so that I can use it from common/protos/multiboot2.c - it works. I hope I can find any insights

mintsuki commented

In that case you could've built Limine with E9_OUTPUT=true passed to make (do a make clean first).

phip1611 commented

Please have a look at #358 - the provided log clearly shows that there is a bug

mintsuki commented

Yes but that's an issue with the printf function we use for testing (which barely works for 32-bit code which this is, since it was made originally as a quick and dirty 64-bit code printf), other than that, the actual values of the memory map look good to me.

The reason they don't look good to you is that the code you posted prints them all at once instead of on several separate printf calls unlike my code.