Feature Request: Add modules to bootx64.efi
ffly90 opened this issue · 64 comments
I am using shredos as an automatic wiping mechanism that boots via pxe and replaces dban because it has efi support.
The bootx64.efi file that comes with shredos has very few modules included, which makes it impossible for me to use it in my setup.
As a workaround I generated a new bootx64.efi file containing the default modules of centos7. With the new generated file I could perform a network boot and autostart the shredding via kernel parameters.
With every Update I need to place the generated file into the folder and other people who are trying to set up a similar scenario, might have the same problem.
The modules that are loaded while performing the pxe boot are the following:
all_video, video_cirrus, video_bochs, efi_uga, efi_gop, net, efinet, tftp, gzio, part_gpt, ext2 (and their dependencies)
I do understand that adding more modules will also increase the size of the image file, but it would make life easier for me and probably others, if you could add those modules to the bootx64.efi that is shipped with shredos.
Thank you for your work!
Yes, you're correct. When I built bootx64.efi some time ago, I only included boot, linux, ext2, fat, squash4, part_msdos, part_gpt, normal, biosdisk, progress, efi_gop, efi_uga for the builtin modules in the buildroot config.
So compared to your list, I'm missing seven additional modules all_video, video_cirrus, video_bochs, net, efinet, tftp, and gzio
I'll run a new build of the bootx64.efi adding those modules and if it boots ok on my systems, I'll do a pre-release that you can test to see if that works, if that's ok?
I'll have a play with that tonight and see how far I get with it.
For future reference, this is a list of available grub2 modules when compiling for x86_efi bootloaders: These modules are available in buildroot but I only build a subset of these modules into bootx64.efi, so that ShredOS builds on the majority of systems. More modules may be added to ShredOS if necessary for specific use cases.
. 15568 Jun 1 21:36 acpi.mod
. 1952 Jun 1 21:36 adler32.mod
. 8096 Jun 1 21:36 affs.mod
. 8336 Jun 1 21:36 afs.mod
. 22912 Jun 1 21:36 ahci.mod
. 840 Jun 1 21:36 all_video.mod
. 1504 Jun 1 21:36 aout.mod
. 5256 Jun 1 21:36 appleldr.mod
. 4592 Jun 1 21:36 archelp.mod
. 8968 Jun 1 21:36 ata.mod
. 6656 Jun 1 21:36 at_keyboard.mod
. 2552 Jun 1 21:36 backtrace.mod
. 9496 Jun 1 21:36 bfs.mod
. 3288 Jun 1 21:36 bitmap.mod
. 5400 Jun 1 21:36 bitmap_scale.mod
. 3032 Jun 1 21:36 blocklist.mod
. 3304 Jun 1 21:36 boot.mod
. 48440 Jun 1 21:36 bsd.mod
. 3320 Jun 1 21:36 bswap_test.mod
. 25240 Jun 1 21:36 btrfs.mod
. 2896 Jun 1 21:36 bufio.mod
. 4440 Jun 1 21:36 cat.mod
. 5784 Jun 1 21:36 cbfs.mod
. 5704 Jun 1 21:36 cbls.mod
. 3976 Jun 1 21:36 cbmemc.mod
. 1640 Jun 1 21:36 cbtable.mod
. 4536 Jun 1 21:36 cbtime.mod
. 9000 Jun 1 21:36 chain.mod
. 4688 Jun 1 21:36 cmdline_cat_test.mod
. 2920 Jun 1 21:36 cmp.mod
. 6576 Jun 1 21:36 cmp_test.mod
. 3544 Jun 1 21:36 configfile.mod
. 4384 Jun 1 21:36 cpio_be.mod
. 4384 Jun 1 21:36 cpio.mod
. 2600 Jun 1 21:36 cpuid.mod
. 2184 Jun 1 21:36 crc64.mod
. 15720 Jun 1 21:36 cryptodisk.mod
. 6928 Jun 1 21:36 crypto.mod
. 3976 Jun 1 21:36 cs5536.mod
. 2696 Jun 1 21:36 ctz_test.mod
. 3064 Jun 1 21:36 datehook.mod
. 3224 Jun 1 21:36 date.mod
. 1880 Jun 1 21:36 datetime.mod
. 13072 Jun 1 21:36 diskfilter.mod
. 3144 Jun 1 21:36 disk.mod
. 1392 Jun 1 21:36 div.mod
. 8096 Jun 1 21:36 div_test.mod
. 2800 Jun 1 21:36 dm_nv.mod
. 3064 Jun 1 21:36 echo.mod
. 2200 Jun 1 21:36 efifwsetup.mod
. 12440 Jun 1 21:36 efi_gop.mod
. 6968 Jun 1 21:36 efinet.mod
. 7160 Jun 1 21:36 efi_uga.mod
. 25768 Jun 1 21:36 ehci.mod
. 6264 Jun 1 21:36 elf.mod
. 2224 Jun 1 21:36 eval.mod
. 8072 Jun 1 21:36 exfat.mod
. 2288 Jun 1 21:36 exfctest.mod
. 8832 Jun 1 21:36 ext2.mod
. 7744 Jun 1 21:36 extcmd.mod
. 9800 Jun 1 21:36 f2fs.mod
. 8168 Jun 1 21:36 fat.mod
. 25896 Jun 1 21:36 file.mod
. 3072 Jun 1 21:36 fixvideo.mod
. 16592 Jun 1 21:36 font.mod
. 4504 Jun 1 21:36 fshelp.mod
. 46688 Jun 1 21:36 functional_test.mod
. 2432 Jun 1 21:36 gcry_arcfour.mod
. 9096 Jun 1 21:36 gcry_blowfish.mod
. 28392 Jun 1 21:36 gcry_camellia.mod
. 14832 Jun 1 21:36 gcry_cast5.mod
. 11928 Jun 1 21:36 gcry_crc.mod
. 16408 Jun 1 21:36 gcry_des.mod
. 3440 Jun 1 21:36 gcry_dsa.mod
. 4048 Jun 1 21:36 gcry_idea.mod
. 4192 Jun 1 21:36 gcry_md4.mod
. 4616 Jun 1 21:36 gcry_md5.mod
. 3152 Jun 1 21:36 gcry_rfc2268.mod
. 20520 Jun 1 21:36 gcry_rijndael.mod
. 7944 Jun 1 21:36 gcry_rmd160.mod
. 3368 Jun 1 21:36 gcry_rsa.mod
. 13352 Jun 1 21:36 gcry_seed.mod
. 16656 Jun 1 21:36 gcry_serpent.mod
. 7768 Jun 1 21:36 gcry_sha1.mod
. 5160 Jun 1 21:36 gcry_sha256.mod
. 6032 Jun 1 21:36 gcry_sha512.mod
. 13144 Jun 1 21:36 gcry_tiger.mod
. 33472 Jun 1 21:36 gcry_twofish.mod
. 22872 Jun 1 21:36 gcry_whirlpool.mod
. 9344 Jun 1 21:36 geli.mod
. 8600 Jun 1 21:36 gettext.mod
. 59696 Jun 1 21:36 gfxmenu.mod
. 4480 Jun 1 21:36 gfxterm_background.mod
. 7816 Jun 1 21:36 gfxterm_menu.mod
. 16920 Jun 1 21:36 gfxterm.mod
. 5360 Jun 1 21:36 gptsync.mod
. 12440 Jun 1 21:36 gzio.mod
. 7496 Jun 1 21:36 halt.mod
. 8496 Jun 1 21:36 hashsum.mod
. 10456 Jun 1 21:36 hdparm.mod
. 1888 Jun 1 21:36 hello.mod
. 3968 Jun 1 21:36 help.mod
. 4472 Jun 1 21:36 hexdump.mod
. 10256 Jun 1 21:36 hfs.mod
. 4296 Jun 1 21:36 hfspluscomp.mod
. 11176 Jun 1 21:36 hfsplus.mod
. 8768 Jun 1 21:36 http.mod
. 4528 Jun 1 21:36 iorw.mod
. 12272 Jun 1 21:36 iso9660.mod
. 8992 Jun 1 21:36 jfs.mod
. 9832 Jun 1 21:36 jpeg.mod
. 6480 Jun 1 21:36 keylayouts.mod
. 3072 Jun 1 21:36 keystatus.mod
. 8232 Jun 1 21:36 ldm.mod
. 45520 Jun 1 21:36 legacycfg.mod
. 15936 Jun 1 21:36 legacy_password_test.mod
. 8672 Jun 1 21:36 linux16.mod
. 19320 Jun 1 21:36 linux.mod
. 4656 Jun 1 21:36 loadbios.mod
. 9176 Jun 1 21:36 loadenv.mod
. 4800 Jun 1 21:36 loopback.mod
. 7160 Jun 1 21:36 lsacpi.mod
. 3712 Jun 1 21:36 lsefimmap.mod
. 5272 Jun 1 21:36 lsefi.mod
. 4248 Jun 1 21:36 lsefisystab.mod
. 2952 Jun 1 21:36 lsmmap.mod
. 6448 Jun 1 21:36 ls.mod
. 7248 Jun 1 21:36 lspci.mod
. 3840 Jun 1 21:36 lssal.mod
. 9800 Jun 1 21:36 luks.mod
. 10016 Jun 1 21:36 lvm.mod
. 7296 Jun 1 21:36 lzopio.mod
. 4936 Jun 1 21:36 macbless.mod
. 10792 Jun 1 21:36 macho.mod
. 2848 Jun 1 21:36 mdraid09_be.mod
. 2816 Jun 1 21:36 mdraid09.mod
. 2680 Jun 1 21:36 mdraid1x.mod
. 3232 Jun 1 21:36 memdisk.mod
. 4544 Jun 1 21:36 memrw.mod
. 5864 Jun 1 21:36 minicmd.mod
. 5768 Jun 1 21:36 minix2_be.mod
. 5664 Jun 1 21:36 minix2.mod
. 5848 Jun 1 21:36 minix3_be.mod
. 5776 Jun 1 21:36 minix3.mod
. 5672 Jun 1 21:36 minix_be.mod
. 5576 Jun 1 21:36 minix.mod
. 9544 Jun 1 21:36 mmap.mod
. 3400 Jun 1 21:36 morse.mod
. 43248 Jun 1 21:36 mpi.mod
. 3736 Jun 1 21:36 msdospart.mod
. 2456 Jun 1 21:36 mul_test.mod
. 24064 Jun 1 21:36 multiboot2.mod
. 21000 Jun 1 21:36 multiboot.mod
. 7344 Jun 1 21:36 nativedisk.mod
. 76816 Jun 1 21:36 net.mod
. 4552 Jun 1 21:36 newc.mod
. 9928 Jun 1 21:36 nilfs2.mod
. 167864 Jun 1 21:36 normal.mod
. 5704 Jun 1 21:36 ntfscomp.mod
. 15232 Jun 1 21:36 ntfs.mod
. 4384 Jun 1 21:36 odc.mod
. 2280 Jun 1 21:36 offsetio.mod
. 15864 Jun 1 21:36 ohci.mod
. 2360 Jun 1 21:36 part_acorn.mod
. 2672 Jun 1 21:36 part_amiga.mod
. 3040 Jun 1 21:36 part_apple.mod
. 4264 Jun 1 21:36 part_bsd.mod
. 2688 Jun 1 21:36 part_dfly.mod
. 2248 Jun 1 21:36 part_dvh.mod
. 3304 Jun 1 21:36 part_gpt.mod
. 3064 Jun 1 21:36 part_msdos.mod
. 2568 Jun 1 21:36 part_plan.mod
. 2280 Jun 1 21:36 part_sun.mod
. 2568 Jun 1 21:36 part_sunpc.mod
. 7360 Jun 1 21:36 parttool.mod
. 2976 Jun 1 21:36 password.mod
. 4560 Jun 1 21:36 password_pbkdf2.mod
. 7400 Jun 1 21:36 pata.mod
. 1944 Jun 1 21:36 pbkdf2.mod
. 3472 Jun 1 21:36 pbkdf2_test.mod
. 3576 Jun 1 21:36 pcidump.mod
. 19320 Jun 1 21:36 pgp.mod
. 3984 Jun 1 21:36 play.mod
. 10464 Jun 1 21:36 png.mod
. 2304 Jun 1 21:36 priority_queue.mod
. 4504 Jun 1 21:36 probe.mod
. 3864 Jun 1 21:36 procfs.mod
. 3192 Jun 1 21:36 progress.mod
. 2000 Jun 1 21:36 raid5rec.mod
. 3312 Jun 1 21:36 raid6rec.mod
. 3680 Jun 1 21:36 random.mod
. 3160 Jun 1 21:36 rdmsr.mod
. 2320 Jun 1 21:36 read.mod
. 1648 Jun 1 21:36 reboot.mod
. 77512 Jun 1 21:36 regexp.mod
. 13856 Jun 1 21:36 reiserfs.mod
. 26408 Jun 1 21:36 relocator.mod
. 5744 Jun 1 21:36 romfs.mod
. 7056 Jun 1 21:36 scsi.mod
. 4768 Jun 1 21:36 search_fs_file.mod
. 4880 Jun 1 21:36 search_fs_uuid.mod
. 4840 Jun 1 21:36 search_label.mod
. 5512 Jun 1 21:36 search.mod
. 14728 Jun 1 21:36 serial.mod
. 1048 Jun 1 21:36 setjmp.mod
. 2664 Jun 1 21:36 setjmp_test.mod
. 8368 Jun 1 21:36 setpci.mod
. 7880 Jun 1 21:36 sfs.mod
. 3176 Jun 1 21:36 shift_test.mod
. 8792 Jun 1 21:36 signature_test.mod
. 3320 Jun 1 21:36 sleep.mod
. 3344 Jun 1 21:36 sleep_test.mod
. 3248 Jun 1 21:36 spkmodem.mod
. 10056 Jun 1 21:36 squash4.mod
. 3288 Jun 1 21:36 strtoull_test.mod
. 29976 Jun 1 21:36 syslinuxcfg.mod
. 5008 Jun 1 21:36 tar.mod
. 6784 Jun 1 21:36 terminal.mod
. 20592 Jun 1 21:36 terminfo.mod
. 2080 Jun 1 21:36 test_blockarg.mod
. 3864 Jun 1 21:36 testload.mod
. 7536 Jun 1 21:36 test.mod
. 3568 Jun 1 21:36 testspeed.mod
. 7424 Jun 1 21:36 tftp.mod
. 7016 Jun 1 21:36 tga.mod
. 2400 Jun 1 21:36 time.mod
. 6104 Jun 1 21:36 tpm.mod
. 2088 Jun 1 21:36 trig.mod
. 3720 Jun 1 21:36 tr.mod
. 1920 Jun 1 21:36 true.mod
. 12264 Jun 1 21:36 udf.mod
. 7904 Jun 1 21:36 ufs1_be.mod
. 7816 Jun 1 21:36 ufs1.mod
. 7848 Jun 1 21:36 ufs2.mod
. 10216 Jun 1 21:36 uhci.mod
. 5872 Jun 1 21:36 usb_keyboard.mod
. 15728 Jun 1 21:36 usb.mod
. 11000 Jun 1 21:36 usbms.mod
. 2880 Jun 1 21:36 usbserial_common.mod
. 3480 Jun 1 21:36 usbserial_ftdi.mod
. 3808 Jun 1 21:36 usbserial_pl2303.mod
. 2392 Jun 1 21:36 usbserial_usbdebug.mod
. 5600 Jun 1 21:36 usbtest.mod
. 8520 Jun 1 21:36 video_bochs.mod
. 9096 Jun 1 21:36 video_cirrus.mod
. 10096 Jun 1 21:36 video_colors.mod
. 28520 Jun 1 21:36 video_fb.mod
. 5344 Jun 1 21:36 videoinfo.mod
. 8904 Jun 1 21:36 video.mod
. 3776 Jun 1 21:36 videotest_checksum.mod
. 5656 Jun 1 21:36 videotest.mod
. 2456 Jun 1 21:36 wrmsr.mod
. 10400 Jun 1 21:36 xfs.mod
. 42264 Jun 1 21:36 xnu.mod
. 3312 Jun 1 21:36 xnu_uuid.mod
. 3232 Jun 1 21:36 xnu_uuid_test.mod
. 19760 Jun 1 21:36 xzio.mod
. 8536 Jun 1 21:36 zfscrypt.mod
. 10696 Jun 1 21:36 zfsinfo.mod
. 57544 Jun 1 21:36 zfs.mod
. 79376 Jun 1 21:36 zstd.mod
@ffly90 I've built the bootx64.efi with the extra modules you needed and it boots fine on a couple of my UEFI test systems. The size of bootx64.efi has increased from 516 KiB to 628 KiB so about 112 KiB increase in size, so not really too much of an increase.
I want to modify the launcher script to fix issue #144 sometime tomorrow and I'll then publish a pre-release that includes the new bootx64.efi and fixes #144
@ffly90 Just out of interest what PXE network server are you running to boot ShredOS? I only ask as I keep thinking I must setup something to test ShredOS with PXE. netboot.xyz as suggested by @Firminator in another post looks like one option, just wondered what others were using.
Just a quick update, I've fixed the --autopoweroff that was causing the logs to not be written to USB. I've also added the dmesg output to the USB, useful for debugging, especially video issues. I've also added a feature that allows loop devices to be created automatically when /dev/loop0 or /dev/loop1 is specified in the nwipe_options in grub.cfg i.e `nwipe_options="--method=zero --autopoweroff /dev/loop0 /dev/loop1 /dev/sda". ShredOS will now create those two loop devices automatically as 1Mbyte drives. This is really for devs or people that want to test ShredOS/nwipe functionality without having to wipe normal drives, more of a time saving feature for devs & testers. And of course includes the updated bootx64.efi with the extra modules.
As soon as the current test passes with no issues, I'll submit the commits and do a pre-release.
Modules added by this commit 98de170. Available in this pre-release v2021.08.2_23.3_x86-64_0.34
@ffly90 If you can give that pre-release a try and let me know if it resolves the problem you were having. I'll close this issue for now, please feel free to reopen the issue should you still have a problem with the latest pre-release above.
I am a co-worker of ffly90. We tested the preview release on our setup and I can report that the PXE stuff does NOT work. bootx64.efi is loaded via tftp but it then fails to find the grub.cfg because it only tries to load it from the built-in $prefix. So the only thing you get is a minimal grub shell. Here's the log output from the tftp server:
Jun 14 17:08:26 server in.tftpd[32650]: RRQ from 192.168.11.11 filename cluster/shredos/bootx64.efi
Jun 14 17:08:26 server in.tftpd[32650]: Client 192.168.11.11 finished cluster/shredos/bootx64.efi
Jun 14 17:08:26 server in.tftpd[32651]: RRQ from 192.168.11.11 filename /EFI/BOOT/x86_64-efi/command.lst
Jun 14 17:08:26 server in.tftpd[32651]: Client 192.168.11.11 File not found /EFI/BOOT/x86_64-efi/command.lst
Jun 14 17:08:26 server in.tftpd[32652]: RRQ from 192.168.11.11 filename /EFI/BOOT/x86_64-efi/fs.lst
Jun 14 17:08:26 server in.tftpd[32652]: Client 192.168.11.11 File not found /EFI/BOOT/x86_64-efi/fs.lst
Jun 14 17:08:26 server in.tftpd[32653]: RRQ from 192.168.11.11 filename /EFI/BOOT/x86_64-efi/crypto.lst
Jun 14 17:08:26 server in.tftpd[32653]: Client 192.168.11.11 File not found /EFI/BOOT/x86_64-efi/crypto.lst
Jun 14 17:08:26 server in.tftpd[32654]: RRQ from 192.168.11.11 filename /EFI/BOOT/x86_64-efi/terminal.lst
Jun 14 17:08:26 server in.tftpd[32654]: Client 192.168.11.11 File not found /EFI/BOOT/x86_64-efi/terminal.lst
Jun 14 17:08:26 server in.tftpd[32655]: RRQ from 192.168.11.11 filename /EFI/BOOT/grub.cfg
Jun 14 17:08:26 server in.tftpd[32655]: Client 192.168.11.11 File not found /EFI/BOOT/grub.cfg
I am not sure what module is missing here but I played around and found out two things:
- it works properly with the (unmodifed) grub64.efi that comes with centos7:
$ rpm -qf /boot/efi/EFI/centos/grubx64.efi
grub2-efi-x64-2.02-0.87.0.2.el7.centos.11.x86_64
$ cp /boot/efi/EFI/centos/grubx64.efi /tftpboot/cluster/shredos/bootx64.efi
$ chmod a+r /tftpboot/cluster/shredos/bootx64.efi
- building an own bootx64.efi on centos7 with a selected number of modules produces a working bootx64.efi:
$ grub2-mkimage -v -p dummy -O x86_64-efi -o bootx64.efi normal linux video video_bochs video_cirrus video_fb efi_gop efi_uga all_video net efinet tftp gzio part_gpt
(Update: Above command line used to include "new" instead of "net"; also "ext2" was missing.. These were just typos/copy'n'paste mistakes that I have now corrected.)
(the -p parameter can be set to anything in our case, it is just a required option. The resulting bootx64.efi will try to load the grub.cfg from there (but fail) and then try the correct location (the directory where bootx64.efi resides). If this bootx64.efi should be used for other purposes (like local boot) you might need to adapt -p to a proper value like /EFI/BOOT that you seem to be using currently).
The grub2-mkimage will include these modules which seem to be sufficient:
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/gettext.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/boot.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/extcmd.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/bufio.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/crypto.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/terminal.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/datetime.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/priority_queue.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/net.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/normal.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/mmap.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/relocator.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/video.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/linux.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/video_fb.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/video_bochs.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/video_cirrus.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/efi_gop.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/efi_uga.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/all_video.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/efinet.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/tftp.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/gzio.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/part_gpt.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/fshelp.mod.
grub2-mkimage: info: reading /usr/lib/grub/x86_64-efi/ext2.mod.
According to your list posted above your build contains some more modules (+dependencies): fat squash4 part_msdos biosdisk progress. So I also added them and it still worked. The only difference is that centos7 comes without the biosdisk module so I could not test if this is the culprit.
PS: we are using dhcp-4.2.5-83.el7.centos.1.x86_64 and tftp-server-5.2-22.el7.x86_64 (via xinetd) on centos7.9
@uli42 That information is really useful. Thanks.
They way I'm currently building the .efi relies upon buildroot to generate the efi file, it gives me the ability to specify modules but that's pretty much it from what I can see. I'm not sure but I think it might rely upon a method that's no longer maintained, possibly https://www.kraxel.org/repos/jenkins/ however this may have changed in the more recent buildroot releases.
I think the easiest thing for me to do to resolve this is to edit ShredOS's doimage.sh script and build the efi as you have suggested above with grub2-mkimage rather than use buildroot to generate the efi.
Testing this on ubuntu 22.04 I can build the efi with the modules you suggest using grub-mkimage v2.06. (I believe that's grub2 even though the command is grub-mkimage, I believe it's grub2 as grub2 version starts at version 1.99. The only difference is new.mod does not exist on my install, however the newc module does. Do you mean newc in your grub2-mkimage command in your previous comment rather than new?
So the modules in my built look like this, which looks like it matches your list.
grub-mkimage: info: writing 704 bytes of a fixup block starting at 0x10000.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/terminal.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/gettext.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/bufio.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/boot.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/datetime.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/priority_queue.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/net.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/crypto.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/extcmd.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/normal.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/mmap.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/video.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/relocator.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/linux.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/video_fb.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/video_bochs.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/video_cirrus.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/efi_gop.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/efi_uga.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/all_video.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/archelp.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/newc.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/efinet.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/tftp.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/gcry_crc.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/gzio.mod.
grub-mkimage: info: reading /usr/lib/grub/x86_64-efi/part_gpt.mod.
grub-mkimage: info: kernel_img=0x7fd6d998b010, kernel_size=0x1c000.
grub-mkimage: info: the core size is 0xa03d0.
grub-mkimage: info: writing 0xa3000 bytes.
The efi file size is 667648 bytes.
Unfortunately I won't be able to test this for a couple of weeks as I've not got access to any of my test kit, but I'll install this grub2 generated efi into the next pre-release. If you could test it out that would be appreciated.
PS: we are using dhcp-4.2.5-83.el7.centos.1.x86_64 and tftp-server-5.2-22.el7.x86_64 (via xinetd) on centos7.9
Thanks, I may setup a PC with this config for testing.
This is a typo! I have not specified a module named "new" but "net". I have no idea how it became "new" in my previous posting! So please retry with "net" and "ext2" (which was missing above, too). I have now also added "progress" since it is really helpful to see the download progress (I had hangs in another situation that would have been far more obvious if I had known about the progress module back then...)
Your output looks good, so once you have re-generated it I am happy to test. It would be sufficient to just provide the bootx64.efi since this is sufficient for testing the pxe part. Unfortunately I cannot build shredos myself due to infrastructure problems (proxy is mandatory but will not properly download the buildroot stuff - don't ask...)
@uli42 I'm finding that I need to add one more module linuxefi for the grub-mkimage created .efi file to boot ShredOS successfully via USB. Without linuxefi it complains about a missing linuxefi.mod file and you just end up at the grub menu.
I'll post the new bootx64.efi for testing later today.
@uli42 Attached is the new bootx64.efi. It was build with the following command:
grub-mkimage -v -p /EFI/BOOT -O x86_64-efi -o bootx64.efi all_video efi_gop efinet efi_uga ext2 fat gzio linux linuxefi net normal part_gpt part_msdos progress squash4 tftp video video_bochs video_cirrus video_fb
It's necessary to specify the prefix -p /EFI/BOOT so that grub loads the grub config /EFI/BOOT/grub.cfg. If I use a dummy prefix ShredOS won't won't boot from USB. Hopefully this prefix won't cause issues for your network boot?
I have tested on a only one laptop at the moment as I don't have access to the rest of my test systems.
The sha1 is 2a207d3d77a0ddb6da19817f0ce87deb0b64c28a bootx64.efi
filesize 737280 bytes
It was necessary to zip the file in order to add it to this comment.
I have tested that - unfortunately it still did not work:
Jun 19 10:13:26 server in.tftpd[22503]: RRQ from 192.168.11.11 filename cluster/shredos/bootx64.efi
Jun 19 10:13:26 server in.tftpd[22503]: Client 192.168.11.11 finished cluster/shredos/bootx64.efi
Jun 19 10:13:26 server in.tftpd[22504]: RRQ from 192.168.11.11 filename cluster/shredos/bootx64.efi
Jun 19 10:13:26 server in.tftpd[22504]: Client 192.168.11.11 finished cluster/shredos/bootx64.efi
[MAC-derived requests removed]
Jun 19 10:13:26 server in.tftpd[22514]: RRQ from 192.168.11.11 filename /EFI/BOOT/x86_64-efi/command.lst
Jun 19 10:13:26 server in.tftpd[22514]: Client 192.168.11.11 File not found /EFI/BOOT/x86_64-efi/command.lst
Jun 19 10:13:26 server in.tftpd[22515]: RRQ from 192.168.11.11 filename /EFI/BOOT/x86_64-efi/fs.lst
Jun 19 10:13:26 server in.tftpd[22515]: Client 192.168.11.11 File not found /EFI/BOOT/x86_64-efi/fs.lst
Jun 19 10:13:26 server in.tftpd[22516]: RRQ from 192.168.11.11 filename /EFI/BOOT/x86_64-efi/crypto.lst
Jun 19 10:13:26 server in.tftpd[22516]: Client 192.168.11.11 File not found /EFI/BOOT/x86_64-efi/crypto.lst
Jun 19 10:13:26 server in.tftpd[22517]: RRQ from 192.168.11.11 filename /EFI/BOOT/x86_64-efi/terminal.lst
Jun 19 10:13:26 server in.tftpd[22517]: Client 192.168.11.11 File not found /EFI/BOOT/x86_64-efi/terminal.lst
Jun 19 10:13:26 server in.tftpd[22518]: RRQ from 192.168.11.11 filename /EFI/BOOT/grub.cfg
Jun 19 10:13:26 server in.tftpd[22518]: Client 192.168.11.11 File not found /EFI/BOOT/grub.cfg
So the only difference between your version and mine is that mine additionally checks the dir where it gt bootx64.efi from. I have tested by build with -p /EFI/BOOT and it still worked for me. So I think it is not tied to the modules but something else.
Centos has added 486 patches to its build - maybe some of them would help. My favourites are these: https://git.centos.org/rpms/grub2/blob/c7/f/SOURCES/0090-Add-fw_path-variable-revised.patch and https://git.centos.org/rpms/grub2/blob/c7/f/SOURCES/0131-use-fw_path-prefix-when-fallback-searching-for-grub-.patch
However, it is worth noting that I don't see the fw_path being set when booting with your bootx64.efi but I see it when booting with the one provided by Centos or one that I build there (with grub2-mkimage). So maybe it is simply this patch that is missing: https://git.centos.org/rpms/grub2/blob/c7/f/SOURCES/0291-efi-http-Export-fw-http-_path-variables-to-make-them.patch
I am really wondering why there are that many patches that do not seem to be included upstream...
What version of grub are you using to create the .efi?
If I remember correctly I was using 2.06 which I think is the latest. Don't quote me on that though.
Can you give me more info on how you setup and configure your PXE server, including versions of centos you are running. If I can setup the same thing then I can hopefully figure the issue out from this end. I know very little about the process of network booting so it will be good for me to setup a test system just like yours a try and get it going.
Sorry, just spotted your previous post. I think we're both thinking along the same lines. I'll take a look at those patches.
One more thing I just noticed:
The working bootx64.efi do contain the string =fw_path= while your build does not:
$ strings grubx64.efi | grep fw_path
fw_path
fw_path:"%s"
fw_path
fw_path
$ strings bootx64.efi | grep fw_path
$
So I think you can use this as basic test.
Of course, the simplest thing for me to do is just setup a VM on my build system with centos and just build the efi on centos in future.
Regarding the the PXE setup:
$ grep -v ^# /etc/xinetd.d/tftp
service tftp
{
disable = no
socket_type = dgram
protocol = udp
wait = yes
user = root
server = /usr/sbin/in.tftpd
server_args = -s /tftpboot -c --verbose
log_on_success += DURATION USERID
log_on_failure += USERID
per_source = 11
cps = 100 2
flags = IPv4
}
dhcpd.conf excerpt:
...
option architecture-type code 93 = unsigned integer 16;
...
subnet 192.168.0.0 netmask 255.255.0.0 {
authoritative;
option broadcast-address 192.168.255.255;
option subnet-mask 255.255.0.0;
pool {
range 192.168.200.1 192.168.200.254;
default-lease-time 1800;
max-lease-time 86400;
option domain-name "ourdomain";
option domain-name-servers 192.168.xx.aa, 192.168.xx.bb;
allow unknown-clients;
next-server 192.168.xx.yy;
# UEFI check
if option architecture-type = 00:07 {
filename "cluster/bootx64.efi";
}
else if option architecture-type = 00:08 {
filename "cluster/bootx64.efi";
}
else if option architecture-type = 00:09 {
filename "cluster/bootx64.efi";
} else {
filename "cluster/pxelinux.0";
}
}
}
...
group { # SHREDOS group
max-lease-time 2592000;
default-lease-time 2592000;
next-server 192.168.xx.yy;
if option architecture-type = 00:07 {
filename "cluster/shredos/bootx64.efi";
}
else if option architecture-type = 00:08 {
filename "cluster/shredos/bootx64.efi";
}
else if option architecture-type = 00:09 {
filename "cluster/shredos/bootx64.efi";
} else {
option magic f1:00:74:7e;
if exists dhcp-parameter-request-list {
option dhcp-parameter-request-list = concat(option dhcp-parameter-request-list,d1);
}
# config file to boot for pxelinux.0 (legacy)
option configfile "shredos/legacy-grub.cfg";
filename "cluster/pxelinux.0";
}
host servera { hardware ethernet 11:22:33:44:55:66; fixed-address 192.168.11.11; option host-name "servera"; }
}
...
So this section will only trigger for the client with MAC 11:22:33:44:55:66.
All the paths in dhcpd.conf a relative to /tftpboot.
$ rpm -q dhcp xinetd tftp-server
dhcp-4.2.5-83.el7.centos.1.x86_64
xinetd-2.3.15-14.el7.x86_64
tftp-server-5.2-22.el7.x86_64
I think that's basically all that's required. (We also have soem firewall rules to allow all this to work but for testing simply disable the firewall).
I think that adding a vm just for this feature is too much, especially for people trying to build their own shredos image. Adding another patch file to the list in =boot/grub2= should much simpler and allow people to build shredos
Yes, you are right. However I'm not sure that buildroot uses grub2-mkimage to generate the efi so I will need to patch grub2 then run the grub2-mkimage command in the ShredOS doimage.sh script to build the efi.
I don't know if you want to submit a pull request with the patches? If not then I'll make the changes in the next week or so.
Regarding the the PXE setup:
Thanks, much appreciated, I'll have a go at setting that up.
Centos has added 486 patches to its build - maybe some of them would help. My favourites are these:..
That's a lot of patches, I wonder why the grub maintainer/s aren't committing them upstream? Maybe just not man power or enough hours in the day to validate them all.
Thanks for letting me know your favourites, I'll look at those patches first, add them and see if that fixes the issue.
I'm wondering whether Ubuntu are also patching grub extensively.
Does sound like a path issue finding grub.cfg.
@uli42 @ffly90 I've got it working with the latest .efi
I configured a tft/dhcp server and booted a client via PXE. The ShredOS efi loads and runs the grub2 code but like you were finding that I just ended up with the grub command line appearing. It appears not to find grub.cfg so the ShredOS kernel never gets loaded. This was with the original ShredOS bootx64.efi, I noted that the prefix and root grub environment variables were empty. I used the set command on it's own from the grub menu to show all the environment values.
I played about with grub.cfg (I'll let you know the details) then copied the latest 700k bootx64.efi into /srv/tftp/EFI/BOOT/ and rebooted the client using PXE, and it loads ShredOS with no issues. As it's a bit late, I'll post more details tomorrow about my tftpd and dhcpd configuration.
@uli42 @ffly90
In order to add legacy booting to the procedure at #148 I was just looking at your dhcpd config described above, where you determine whether we are booting UEFI or legacy. Extract shown below:
if option architecture-type = 00:07 {
filename "cluster/shredos/bootx64.efi";
}
else if option architecture-type = 00:08 {
filename "cluster/shredos/bootx64.efi";
}
else if option architecture-type = 00:09 {
filename "cluster/shredos/bootx64.efi";
} else {
option magic f1:00:74:7e;
if exists dhcp-parameter-request-list {
option dhcp-parameter-request-list = concat(option dhcp-parameter-request-list,d1);
}
# config file to boot for pxelinux.0 (legacy)
option configfile "shredos/legacy-grub.cfg";
filename "cluster/pxelinux.0";
}
There's a few things I don't understand.
- What does this do? where is this list and what's in it. What is d1?
option dhcp-parameter-request-list = concat(option dhcp-parameter-request-list,d1);
- Whats is
option magic f1:00:74:7e; - Where does this filename
filename "cluster/pxelinux.0";come from?
I would like to add legacy PXE build to #148 but don't understand those parts of the config.
Oh, one more question: :-)
- What's the relevance of
architecture-type = 00:07,architecture-type = 00:08andarchitecture-type = 00:09when related to hardware. What's the difference between each architecture type?
After a bit of googling it looks like pxelinux.0 is obtained from the syslinux package?
Edit: In ubuntu it's in the pxelinux package.
No need, I got a basic PXE server running using a slightly modified version of the dhcpd.conf which boots ShredOS fine via Shredos's bootx64.efi and on a non efi system boots pxelinux although hangs after running the pxelinux code, not sure why but then I came across grub2 with PXE which I like the look of.
So I'm now looking at creating a pxe server with grub2/pxe rather than syslinux/pxelinux for both UEFI & legacy with multiple boot options.
@uli42 @ffly90 In order to add legacy booting to the procedure at #148 I was just looking at your dhcpd config described above, where you determine whether we are booting UEFI or legacy. Extract shown below:
if option architecture-type = 00:07 { filename "cluster/shredos/bootx64.efi"; } else if option architecture-type = 00:08 { filename "cluster/shredos/bootx64.efi"; } else if option architecture-type = 00:09 { filename "cluster/shredos/bootx64.efi"; } else { option magic f1:00:74:7e; if exists dhcp-parameter-request-list { option dhcp-parameter-request-list = concat(option dhcp-parameter-request-list,d1); } # config file to boot for pxelinux.0 (legacy) option configfile "shredos/legacy-grub.cfg"; filename "cluster/pxelinux.0"; }There's a few things I don't understand.
- What does this do? where is this list and what's in it. What is d1?
option dhcp-parameter-request-list = concat(option dhcp-parameter-request-list,d1);
https://linux.die.net/man/5/dhcpd-options
"This option, when sent by the client, specifies which options the client wishes the server to return. [...] When this option is specified on the server, the server returns the specified options. This can be used to force a client to take options that it hasn't requested, and it can also be used to tailor the response of the DHCP server for clients that may need a more limited set of options than those the server would normally return."
So here it means the server should specifically return option 209 (0xd1) which is the config file to load. This is specific for pxelinux which is only for legacy systems, not for UEFI.
- Whats is
option magic f1:00:74:7e;
https://www.rfc-editor.org/rfc/rfc5071.html#page-4
"If this option is provided to the PXE bootloader, then the value is
checked by PXELINUX to match the octet string f1:00:74:7e. If this
matches, then PXELINUX bootloaders will also consume options 209-211,
as described below. Otherwise, they are ignored."
According to https://techdirectarchive.com/2020/05/18/information-on-bootp-vendor-extensions-and-dhcp-options/ this is deprecated now. Also it is only relevant for pxelinux and not for uefi boots.
- Where does this filename
filename "cluster/pxelinux.0";come from?
This is the legacy boot file for non-UEFI systems.
I would like to add legacy PXE build to #148 but don't understand those parts of the config.
Explained ;-)
I have tested with the bootx64.efi you posted above on June 17th and with the once included in shredos-2021.08.2_23.5_x86-64_0.34_20230702.img. Both do not work with our PXE setup.
They both try to load the grub.cfg from /EFI/BOOT/grub.cfg. I think this is because the /EFI/BOOT is hardcoded but our dir is not the top-level dir (grub.cfg is actually located at cluster/shredos/EFI/BOOT/grub.cfg). I guess it only works with this bootx64.efi when the according dir structure exists at the top-level tftp dir, not below.
Jul 18 11:48:44 smtcac1510 in.tftpd[10863]: RRQ from 192.168.11.15 filename cluster/shredos/EFI/BOOT/bootx64.efi
Jul 18 11:48:44 smtcac1510 in.tftpd[10863]: Error code 8: User aborted the transfer
Jul 18 11:48:44 smtcac1510 in.tftpd[10864]: RRQ from 192.168.11.15 filename cluster/shredos/EFI/BOOT/bootx64.efi
Jul 18 11:48:44 smtcac1510 in.tftpd[10864]: Client 192.168.11.15 finished cluster/shredos/EFI/BOOT/bootx64.efi
Jul 18 11:48:44 smtcac1510 in.tftpd[10866]: RRQ from 192.168.11.15 filename cluster/shredos/EFI/BOOT/bootx64.efi
Jul 18 11:48:44 smtcac1510 in.tftpd[10866]: Client 192.168.11.15 finished cluster/shredos/EFI/BOOT/bootx64.efi
Jul 18 11:48:51 smtcac1510 in.tftpd[10873]: RRQ from 192.168.11.15 filename /EFI/BOOT/x86_64-efi/command.lst
Jul 18 11:48:51 smtcac1510 in.tftpd[10873]: Client 192.168.11.15 File not found /EFI/BOOT/x86_64-efi/command.lst
Jul 18 11:48:51 smtcac1510 in.tftpd[10874]: RRQ from 192.168.11.15 filename /EFI/BOOT/x86_64-efi/fs.lst
Jul 18 11:48:51 smtcac1510 in.tftpd[10874]: Client 192.168.11.15 File not found /EFI/BOOT/x86_64-efi/fs.lst
Jul 18 11:48:51 smtcac1510 in.tftpd[10875]: RRQ from 192.168.11.15 filename /EFI/BOOT/x86_64-efi/crypto.lst
Jul 18 11:48:51 smtcac1510 in.tftpd[10875]: Client 192.168.11.15 File not found /EFI/BOOT/x86_64-efi/crypto.lst
Jul 18 11:48:51 smtcac1510 in.tftpd[10876]: RRQ from 192.168.11.15 filename /EFI/BOOT/x86_64-efi/terminal.lst
Jul 18 11:48:51 smtcac1510 in.tftpd[10876]: Client 192.168.11.15 File not found /EFI/BOOT/x86_64-efi/terminal.lst
Jul 18 11:48:51 smtcac1510 in.tftpd[10877]: RRQ from 192.168.11.15 filename /EFI/BOOT/grub.cfg
Jul 18 11:48:51 smtcac1510 in.tftpd[10877]: Client 192.168.11.15 File not found /EFI/BOOT/grub.cfg
(you see, /EFI/BOOT is relative to the top-level tftp dir, not to <top-level>/cluster/shredos)
According to an answer I got on the help-grul ML it should work to load the grub.cfg from the same location as the bootx64.efi when the path is set to "". So it would be great if you could provide such a build.
I have patched the bootx64.efi from the shredos-2021.08.2_23.5_x86-64_0.34_20230702.img to have an empty search path like this:
$ strings -t d bootx64.efi.orig | grep /EFI/BOOT
637792 /EFI/BOOT
$ dd f=/dev/zero of=bootx64.efi bs=1 count=1 seek=637792 conv=notrunc
1+0 records in
1+0 records out
1 byte (1 B) copied, 0.000169346 s, 5.9 kB/s
$ md5sum bootx64.efi
5ce32fae7bd8e34b68944518ed5503f0 bootx64.efi
And then it loads the grub-conf from the right location:
Jul 18 12:20:12 smtcac1510 in.tftpd[14420]: RRQ from 192.168.11.15 filename cluster/shredos/EFI/BOOT/x86_64-efi/command.lst
Jul 18 12:20:12 smtcac1510 in.tftpd[14420]: Client 192.168.11.15 File not found cluster/shredos/EFI/BOOT/x86_64-efi/command.lst
Jul 18 12:20:12 smtcac1510 in.tftpd[14421]: RRQ from 192.168.11.15 filename cluster/shredos/EFI/BOOT/x86_64-efi/fs.lst
Jul 18 12:20:12 smtcac1510 in.tftpd[14421]: Client 192.168.11.15 File not found cluster/shredos/EFI/BOOT/x86_64-efi/fs.lst
Jul 18 12:20:12 smtcac1510 in.tftpd[14422]: RRQ from 192.168.11.15 filename cluster/shredos/EFI/BOOT/x86_64-efi/crypto.lst
Jul 18 12:20:12 smtcac1510 in.tftpd[14422]: Client 192.168.11.15 File not found cluster/shredos/EFI/BOOT/x86_64-efi/crypto.lst
Jul 18 12:20:12 smtcac1510 in.tftpd[14423]: RRQ from 192.168.11.15 filename cluster/shredos/EFI/BOOT/x86_64-efi/terminal.lst
Jul 18 12:20:12 smtcac1510 in.tftpd[14423]: Client 192.168.11.15 File not found cluster/shredos/EFI/BOOT/x86_64-efi/terminal.lst
Jul 18 12:20:12 smtcac1510 in.tftpd[14424]: RRQ from 192.168.11.15 filename cluster/shredos/EFI/BOOT/grub.cfg
Jul 18 12:20:12 smtcac1510 in.tftpd[14424]: Client 192.168.11.15 finished cluster/shredos/EFI/BOOT/grub.cfg
Jul 18 12:20:12 smtcac1510 in.tftpd[14425]: RRQ from 192.168.11.15 filename /boot/shredos
Jul 18 12:20:12 smtcac1510 in.tftpd[14425]: Client 192.168.11.15 File not found /boot/shredos
Tha final too lines are expected as I still have the original grub.conf lying around. Maybe we can also find a way to make the very same file work for pen drive boots and PXE boots.
Success: When changing grub.cfg to something like this it works with PXE and should also work when booted from pen drive:
menuentry "shredos" {
linux ${cmdpath}/../../boot/shredos console=tty3 loglevel=3
}
or
menuentry "shredos" {
linux ${prefix}/../../boot/shredos console=tty3 loglevel=3
}
If the changes you have made to grub.cfg works with both PXE and pen drives, I can make the suggested changes. I won't be able to test whether it works with pen drives myself this week as I'm focused on finishing the PDF code for the upcoming release, but will test it next week when I get a chance.
That explains why my simple PXE server setup #148 worked fine as shredos was copied to the root of the tftp directory /srv/tftp/boot/shredos
Exactly. No need to hurry... ;-)
One more thing: with the current shredos-2021.08.2_23.5_x86-64_0.34_20230702.img bootx64 (patched with dd like above) I see problems downloading the shredos image, the download very often fails with a timeout. With the one from Jun 17 (also patched) it works everytime. So you should definitely use the options you used to build the June 17 version.
That's odd. I've just run sha1sum on bootx64.efi on both the Jun17 version and the one that's in the shredos-2021.08.2_23.5_x86-64_0.34_20230702.img release and they are the exact same file. Both return the checksum 2a207d3d77a0ddb6da19817f0ce87deb0b64c28a bootx64.efi
So that's not the problem, may be worth comparing them with sha1sum after you applied the dd patch to makesure the patched versions should be identical.
Hmm, I have just checked:
2a207d3d77a0ddb6da19817f0ce87deb0b64c28a EFI/BOOT/bootx64.efi_github_issue_147
30c30c184df905c299ad5419e6e2e8b347d3720f EFI/BOOT/bootx64.efi_github_issue_147.patched
097118b7defba918d0ba2ad40a8505c72d2c093e EFI/BOOT/bootx64.efi.orig
3137f25a60260803e3d2223eeb2be3090c5f6ada EFI/BOOT/bootx64.efi.patched
.orig is the one included in shredos-2021.08.2_23.5_x86-64_0.34_20230702.img. So I cannot find a mistake.
Sorry, the mistake is mine, you are correct 23.5 is using the wrong bootx64.efi and not the one from Jun 17. My working build has the Jun17 bootx64.efi which was leading to the confusion.
Before I do the next release I'll makesure the Jun17 efi is definitely included.
ok, great!
Re:
menuentry "shredos" {
linux ${cmdpath}/../../boot/shredos console=tty3 loglevel=3
}
I tested that on a pen drive and it boots fine using EFI, but fails on legacy with file not found or if you prefix with two periods, i.e ../../../boot/shredos console=tty3 loglevel=3 then you get filesystem not found.
So I tried setting up fallback but despite the legacy booting manually from the grub menu it wouldn't boot at startup complaining that it could not boot from the default or fallback entries. If anybody can figure out how to set this up in grub to try the first entry and automatically fall back to the second that would be appreciated.
set default="0"
set timeout="0"
fallback 1
title ShredOS_PXE
menuentry "ShredOS PXE" {
linux ${cmdpath}../../../boot/shredos console=tty3 loglevel=3
}
savedefault fallback
title ShredOS
menuentry "Shredos Standard" {
linux /boot/shredos console=tty3 loglevel=3
}
savedefault
This works ! It puts a prompt up on the screen after it fails the pxe boot then if you don't do anything after about 5 seconds or so it falls back and tried the standard path.
Be nice if it didn't have the prompt, don't know if there is an option to get rid of that.
Also, as most people probably? use a flash drive I would have the entries so standard boot was tried first then falls back to PXE.
set default="0"
set timeout="0"
set fallback="1"
title ShredOS_PXE
menuentry "ShredOS PXE" {
linux ${cmdpath}../../../boot/shredos console=tty3 loglevel=3
}
title ShredOS_Standard
menuentry "ShredOS Standard" {
linux /boot/shredos console=tty3 loglevel=3
}
Are you able to try this out on your PXE server and see if it works?
set default="0"
set timeout="0"
set fallback="1"
title ShredOS_PXE
menuentry "ShredOS PXE" {
linux ${cmdpath}../../../boot/shredos console=tty3 loglevel=3
}
title ShredOS_Standard
menuentry "ShredOS Standard" {
linux /boot/shredos console=tty3 loglevel=3
}
@uli42 Actually, can you try this one. The menu entries are ordered so that the standard boot path is tried first then the PXE path. This would be the way I would prefer it to be ordered in the final release.
set default="0"
set timeout="0"
set fallback="1"
title ShredOS_Standard
menuentry "ShredOS Standard" {
linux /boot/shredos console=tty3 loglevel=3
}
title ShredOS_PXE
menuentry "ShredOS PXE" {
linux ${cmdpath}../../../boot/shredos console=tty3 loglevel=3
}
If anybody can figure out how to set this up in grub to try the first entry and automatically fall back to the second that would be appreciated.
https://askubuntu.com/a/1214284/560368 suggests this:
if test "${grub_platform}" = "pc"; then
menuentry 'XYZ' {
# legacy config
}
else
menuentry 'XYZ' {
#uefi config
}
fi
In grub you can also open the commandline and run =set=. This will show all environment vars, so maybe there's another one instead of $cmdpath that can be used here. In my (PXE UEFI) case $prefix and $cmdline were the same, but maybe on Legacy BIOS systems on $prefix is better.
Edit: for this to work you need to include the "test" module in bootx64.efi. And for better testing also "configfile" and "eval"
@uli42 Actually, can you try this one. The menu entries are ordered so that the standard boot path is tried first then the PXE path. This would be the way I would prefer it to be ordered in the final release.
set default="0" set timeout="0" set fallback="1" title ShredOS_Standard menuentry "ShredOS Standard" { linux /boot/shredos console=tty3 loglevel=3 } title ShredOS_PXE menuentry "ShredOS PXE" { linux ${cmdpath}../../../boot/shredos console=tty3 loglevel=3 }
Hmm, this would show two menu entries, one them being broken. Not very smooth.
I know why ${prefix} doesn't work under legacy, i.e ${prefix} is something like {hd0,msdos1}/boot/grub, which is the default grub location, but ShredOS puts the image in /boot/.
The only way I can get it to work under legacy while including the string "/../../../" is as follows:
- Use the command
linux (hd0,msdos1)/../../../boot/shredos console=tty3 loglevel=3
Notice the forward slash after ${prefix}, it won't boot without that. i can set (hd0,msdos1) as prefix so the command would become linux ${prefix}/../../../boot/shredos console=tty3 loglevel=3
On my legacy system $prefix defaults to (hd0,msdos1)/boot/grub/ which is not the correct path so there is an inconsistency there which needs fixing. It should be (hd0,msdos1)/boot.
Will your PXE system boot using the same command?
i.e linux (hd0,msdos1)/../../../boot/shredos console=tty3 loglevel=3
what's your cmdpath and prefix set as?
@uli42 Actually, can you try this one. The menu entries are ordered so that the standard boot path is tried first then the PXE path. This would be the way I would prefer it to be ordered in the final release.
set default="0" set timeout="0" set fallback="1" title ShredOS_Standard menuentry "ShredOS Standard" { linux /boot/shredos console=tty3 loglevel=3 } title ShredOS_PXE menuentry "ShredOS PXE" { linux ${cmdpath}../../../boot/shredos console=tty3 loglevel=3 }Hmm, this would show two menu entries, one them being broken. Not very smooth.
And not working anyway:
The PXE version is also not working when chosen explicitly. When I remote the ".." following ${cmdpath} it works when chosen directly, but not via fallback.
linux (hd0,msdos1)/../../../boot/shredos console=tty3 loglevel=3
No, this results in "error: disk (hd0,msdos1) not found"
what's your cmdpath and prefix set as?
(as there's no echo and no less or something I cannot show cmdpath in its full length but this should be enough to indicate they are he same:
Just typing set will show all the grub environment variables.
Just typing
setwill show all the grub environment variables.
Yes, I know, but that does not help when there are more that fit on the screen. There's no scrolling, no echo, no less or anything. And just typing the variable name as I did above somehow does not show the full content.
Update: Typos...
I think I need to setup my PXE server using the same directory structure you are using, i.e. cluster/ShredOS/.. I can then try out a few things.
I'm a bit surprised your prefix above contains what looks like the full path to bootx64.efi
That means what ever I use for prefix is replaced by your setup so maybe this command might work after all
linux ${prefix}/../../../boot/shredos console=tty3 loglevel=3 on my legacy system once I've corrected the location of the ShredOS image to /boot/grub/ instead of /boot/ and changing the Linux command to
linux ${prefix}/../../../boot/grub/shredos console=tty3 loglevel=3 so it uses the standard location that other distros use.
Just typing
setwill show all the grub environment variables.Yes, I know, but that does not help when there are more that fit on the screen. There's no scrolling, no echo, no less or anything. And just typing the varaible name as I did above somehow does not shoe the full content.
You would think there was some way of enabling line wrap, but I can't see any grub command for that. There is a echo command in grub, not sure that would help with line wraps though.
I've set up my PXE server with /cluster/shredos/.. and can see the issues, I've not managed to get a working configuration that allows PXE booting and USB with a common config, however I'm now starting to think maybe I need to write a small script similar to this that tests for the various locations and then sets the prefix up once it's found the locations before loading the kernel. Grub Shell like scripting
Just an example, the directories/files aren't relevant to shredos.
search.fs_label grub root
if [ -e /boot/grub/example/test1.cfg ]; then
set prefix=($root)/boot/grub
configfile /boot/grub/example/test1.cfg
else
if [ -e /boot/grub/example/test2.cfg ]; then
set prefix=($root)/boot/grub
configfile /boot/grub/example/test2.cfg
else
echo "Could not find an example configuration file!"
fi
fi
Ubuntu's grub.cfg is a fine example of grub scripting.
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#
### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
set have_grubenv=true
load_env
fi
if [ "${initrdfail}" = 2 ]; then
set initrdfail=
elif [ "${initrdfail}" = 1 ]; then
set next_entry="${prev_entry}"
set prev_entry=
save_env prev_entry
if [ "${next_entry}" ]; then
set initrdfail=2
fi
fi
if [ "${next_entry}" ] ; then
set default="${next_entry}"
set next_entry=
save_env next_entry
set boot_once=true
else
set default="0"
fi
if [ x"${feature_menuentry_id}" = xy ]; then
menuentry_id_option="--id"
else
menuentry_id_option=""
fi
export menuentry_id_option
if [ "${prev_saved_entry}" ]; then
set saved_entry="${prev_saved_entry}"
save_env saved_entry
set prev_saved_entry=
save_env prev_saved_entry
set boot_once=true
fi
function savedefault {
if [ -z "${boot_once}" ]; then
saved_entry="${chosen}"
save_env saved_entry
fi
}
function initrdfail {
if [ -n "${have_grubenv}" ]; then if [ -n "${partuuid}" ]; then
if [ -z "${initrdfail}" ]; then
set initrdfail=1
if [ -n "${boot_once}" ]; then
set prev_entry="${default}"
save_env prev_entry
fi
fi
save_env initrdfail
fi; fi
}
function recordfail {
set recordfail=1
if [ -n "${have_grubenv}" ]; then if [ -z "${boot_once}" ]; then save_env recordfail; fi; fi
}
function load_video {
if [ x$feature_all_video_module = xy ]; then
insmod all_video
else
insmod efi_gop
insmod efi_uga
insmod ieee1275_fb
insmod vbe
insmod vga
insmod video_bochs
insmod video_cirrus
fi
}
........
.......
......
.....
....
...
..
.
@uli42 Your prefix has the full path prefix=(tftp, 192.168.50.253) cluster/shredos/EFI/BOOT to the efi grub.cfg. is that something you have configured? Just wondering as my prefix just has the IP but no path.
The way I'm thinking of doing it it is to only have the prefix contain the IP the rest of the path is created with the grub.cfg depending upon whether /cluster/shedos/boot/shredos exists at that location if it does the standard path is created for booting from hd0/USB. It should then cope with all the boot methods.
@uli42 Your prefix has the full path
prefix=(tftp, 192.168.50.253) cluster/shredos/EFI/BOOTto the efi grub.cfg. is that something you have configured? Just wondering as my prefix just has the IP but no path.
No. The thing is: $prefix and $cmdpath is provided by bootx64.efi, derived from bootx64.efi path, AFAIU. So it is different for my pxe server than for yours or for a USB boot. Because of that I was trying to refererence the shredos binary relative to $prefix, so that it is always found, regardless of the boot method. That way you only have to have one entry that should always work.
The way I'm thinking of doing it it is to only have the prefix contain the IP the rest of the path is created with the grub.cfg depending upon whether /cluster/shedos/boot/shredos exists at that location if it does the standard path is created for booting from hd0/USB. It should then cope with all the boot methods.
For the legacy boot it is possible that the environment is not suited for above "relative to $prefix" confguration. That's why I suggest including the "test" command to have an if clause in grub.conf.
I think I need to setup my PXE server using the same directory structure you are using, i.e. cluster/ShredOS/.. I can then try out a few things.
I'm a bit surprised your prefix above contains what looks like the full path to bootx64.efi
That means what ever I use for prefix is replaced by your setup so maybe this command might work after all
linux ${prefix}/../../../boot/shredos console=tty3 loglevel=3on my legacy system once I've corrected the location of the ShredOS image to /boot/grub/ instead of /boot/ and changing the Linux command tolinux ${prefix}/../../../boot/grub/shredos console=tty3 loglevel=3so it uses the standard location that other distros use.
I am still wondering why you are using three ".." here. $prefix points to EFI/BOOT, so EFI/BOOT/../../boot/shredos should always be correct.
I've set up my PXE server with /cluster/shredos/.. and can see the issues, I've not managed to get a working configuration that allows PXE booting and USB with a common config, however I'm now starting to think maybe I need to write a small script similar to this that tests for the various locations and then sets the prefix up once it's found the locations before loading the kernel. Grub Shell like scripting
...
set prefix=($root)/boot/grub
...
Just a feeling, but I think one should not change prefix as this is something the systems sets accordingly. It should not be necessary, either.
Just a feeling, but I think one should not change prefix as this is something the systems sets accordingly. It should not be necessary, either.
Agreed.
@uli42 I think I've found a solution. It requires an updated ShredOS build for legacy and a new bootx64.efi. This is because both the ShredOS build (for legacy boot) and the bootx64.efi require four extra modules. These are read, eval, test and true. Without these modules the if and test statements don't evaluate correctly and throw up syntax errors.
With the above changes this new grub.cfg now boots both PXE efi and USB efi correctly. It also appears it will work fine for both legacy PXE and USB, although I need to rebuild ShredOS and do more tests.
The new grub.cfg retains the single menuentry but alters its path on the linux command line via the $shredos_path variable. This changes it's value based on whether the file /cluster/shredos/boot/shredos exists or not, so basically if it detects a PXE path it will use that. If not it does a standard boot. ${root} is automatically updated by the PXE server with tftp, IP address and if no PXE it uses the standard hd0,msdos1.
So operates nice and smooth with no fallback. :-)
set default="0"
set timeout="0"
if test -f "/cluster/shredos/boot/shredos"; then
set shredos_path=(${root})/cluster/shredos
else
set shredos_path="(${root})"
fi
menuentry shredos {
linux ${shredos_path}/boot/shredos console=tty3 loglevel=3
}
If you want to test it, let me know and I'll do a full rebuild and pre-release.
Oh, and the bootx64.efi needs to be build with an empty prefix string, so -p "" rather than -p "EFI/BOOT. The ISP-DHCPD-SERVER configuration in /etc/dhcpd.conf already knows about the location of bootx64.efi and the efi grub.cfg files via the entry option bootfile-name "cluster/shredos/EFI/BOOT/bootx64.efi so the previous efi prefix EFI/BOOT simply messes with the order of things and over complicates it. We don't need prefix anyway because as we take care of that with the $root and $shredos_path variables.
But isn't the hardcoded path in grub.conf superfluous and can be replaced by ${prefix}/../..
For PXE, yes but the problem then comes with USB booting, from what I've seen, $prefix only ever seems to contain hd0,msdos1 when booting USB, where as PXE booting prefix contains a path in addition to the tftp,IP_ADDRESS/path so /../.. might be ok for PXE but results in a file not found in USB. $root is predictable, so it's easier to use $root as the base the construct the rest of the path depending on the boot method being employed. I'm definitely no grub expert, but at the moment it's the only way I've managed to get it to reliably boot USB and PXE using the same identical grub.cfg.
Lot's more testing to go before I;m happy with it. I need to boot it on all my test systems which are a mix of UEFI & Legacy and have the ones that support PXE all boot from the server.
I don't know if you can post your /etc/dhcpd.conf file, maybe mine is not setup as good as it should be so having a look at something more complicated may help me understand your config better. I'm just setting up pxelinux.0 so I can boot legacy, I'm guessing your dhcpd.conf may already have legacy and multiple other OS'es too?
It's probably worth checking whether $prefix can replace that path, as thinking about the efi built it was the -p "EFI/BOOT/" that was causing the issues, I think it was getting appended to the end of cluster/shredos/EFI/BOOT and so was looking in the wrong place for the kernel image. There were other inconsistencies with prefix, sometimes it might contain tftp,IP/path or hd0,msdos1 with or without a path so depending on what prefix was being set it was hard to create a consistent path. $root never has anything other than tftp,IP or hd0,msdos1 never any paths, so any paths build on that within grub.cfg I knew was going to be correct.
But, it's certainly worth trying with prefix and seeing if it's reliable across multiple systems.