This project attempts to provide a working example system combining Buildroot, U-Boot, and RAUC in a "works out of the box" example for the Raspberry Pi Compute Module 4 (CM4). The intent is for this to be usable as a base system for some classes of IoT projects, and a hopefully easy to modify starting point if you need something more customized to your needs.
- U-Boot bootloader with redundant environment storage
- Symmetric Root-FS Slots with fallback on failed updates
- Atomic updates of bootloader vfat (no fallback)
- Run-time watchdog timeouts (via systemd)
- Single persistent data partition
- Boot-time watchdog support (work in progress)
This project is a Buildroot external directory and can not be used alone. You also need to have a local copy of the Buildroot project and a properly configured build. One simple way to do this would be:
# Create a working directory
mkdir ~/MyWorkDir
cd ~/MyWorkDir
# Pull in the required projects
git clone --depth 1 --branch 2021.11.x https://git.busybox.net/buildroot/
git clone https://github.com/cdsteinkuehler/br2rauc
# Setup buildroot, keeping build artifacts outside the buildroot tree
# Note paths are relative to the buildroot directory
make -C buildroot/ BR2_EXTERNAL=../br2rauc O=../output raspberrypicm4io_64_rauc_defconfig
cd output
# You can now run standard buildroot make commands with no options
# You may want to run "make menuconfig" to enable ccache or review/tweak the
# default config selections
# Build everything and generate the sdcard.img image file:
makeFor quick experiments, you may wish to modify the br2rauc tree directly, or a more sophisticated process might use a top-level project that includes several exteranl trees and multiple BR2_EXTERNAL entries, eg:
-- MyCoolProject
|-- buildroot (git external)
|-- br2rauc (git external)
|-- MyApp1 (external or part of MyCoolProject)
|-- MyApp2 (external or part of MyCoolProject)
|-- output (generated by Buildroot, can be renamed or moved elsewhere)
cd MyCoolProject
make -C buildroot/ BR2_EXTERNAL=../br2rauc:../MyApp1:../MyApp2 ...After completing a build, your Buildroot output directory should contain an image file (output/images/sdcard.img) that can be written to a uSD card for use with the Raspberry Pi Compute Module 4 IO Board. In addition to the uSD you will need a 3.3V serial cable connected to the standard serial console pins on the cm4io (gpio 14 & 15, pins 8 & 10 on the 40-pin connector).
Once the system boots, you can use the rauc command to examine the current status:
rauc status
...or to mark the current partition as good:
rauc status mark-good
Note that if you do not mark the status as good the boot count in the bootload will count down and eventually switch to the other partition. You can use the fw_printenv command to examine the current bootloader status:
$ sudo fw_printenv | grep BOOT_
BOOT_A_LEFT=2
BOOT_B_LEFT=3
BOOT_ORDER=A B
...and see what happens after marking the current boot as 'good':
$ rauc status mark-good
rauc-Message: 17:34:00.021: rauc status: marked slot rootfs.0 as good
$ sudo fw_printenv | grep BOOT_
BOOT_B_LEFT=3
BOOT_ORDER=A B
BOOT_A_LEFT=3
For full details, read through the RAUC manual.
This project is based on the Buildroot provided 64-bit default configuration for the Raspberry Pi CM4 I/O board, with the following major changes:
- Switch to glibc, systemd, and udev : The RPi is not particularly resouce constrained, and using glibc, systemd, and udev provides the fewest surprises when migrating from a Raspberry Pi OS based system. In particular, if you are not using udev many device nodes will not get properly generated, particularly the video devices
- Switch to the U-Boot boot loader : U-Boot is required to allow intelligent switching between redundant filesystem images and handling failed updates.
- Add RAUC : This builds the required host and target tools needed to use RAUC.
- Undefine BR2_ARM_FPU_VFPV4 : 64-bit ARM cores are required to support ARMv8.
- Add a non-root user (br2rauc) and enable sudo
- Implement device tree customizations necessary for the cm4io
- Modify post-*.sh scripts as needed for RAUC
- Generate RAUC update bundles for boot and root filesystems
The U-Boot configuration leverages the default rpi_arm64 configuation provided by upstream. A configuration fragment file is used to override a few options:
- Store environment on the mmc device
- Enable redundant copies of the environment
The RPi firmware loads and modifies the device tree based on the contents of the config.txt file. The original device-tree file may not boot without some of these changes (eg: the dma-ranges property for the emmc controller is different between the BCM2711 C0T stepping revisions).
The RPi firmware uses more than just cmdline.txt to construct the kernel command line. If you do not duplicate these entries, your system may not boot (eg: the rootwait parameter is required when booting from emmc).
The easiest way to determine exactly what the firmware is doing is to boot using the firmware provided settings (fdt blob at ${fdt_addr} with no bootargs specified by U-Boot), examine the run-time system that results, and compare with the original source files.
The provided U-Boot script will use the firmware provided device tree if the kernel command line includes the text "fw_dtb", otherwise the device tree is loaded from the appropriate rootfs partition and the emmc dma-ranges property is copied from the fixed-up fdt provided by the firmware. This makes it fairly easy to switch between using the RPi firmware to generate a device tree (so you have support for dtoverlay= in config.txt) and a (likely flattened) device tree built along with the kernel. For more details, see the Device Tree section, below.
The Linux kernel configuration leverages the same bcm2711 configuration as the
Buildroot default Raspberry Pi examples. A configuration fragment is used to
enable verity and squashfs, required to work with the new format RAUC bundles.
Since a Raspberry Pi kernel is used, the Linux Kernel version is stored as part of the default config file and will not be updated if you switch to a newer version of Buildroot and rebuild from scratch. Make sure to update the BR2_LINUX_KERNEL_CUSTOM_TARBALL_LOCATION setting in your configuration when updating Buildroot!
Managing the device tree can be one of the more complicated aspects of working with embedded ARM based systems. A full discussion of all available options for managing device trees is well beyond the scope of this project, but a brief list of some possible options includes:
-
Let the RPi firmware load your device tree: This creates a dependency between your rootfs image containing the kernel and the FAT partition with the device tree file and overlays. This is a Very Bad way to handle your device tree for an actual product, but can be useful for development, especially as you transition from a full Raspberry Pi OS development environment to the more streamlined Buildroot environment. To use the device tree loaded by the RPi firmware, pass the fw_dtb argument on the kernel command line. This is currently the default for the generated sdcard.img system.
-
Migrate device tree overlay processing to U-Boot: This is non-trivial, but certainly possible (see the BeagleBone, for example).
-
Manually generate a flattened device tree: This may be a decent intermediate option if you are willing to do some manual work when updating kernel versions. For example, you could boot using the RPi firmware and config.txt file then copy the run-time device tree to a flattened dts file:
dtc -I fs -O dts -s /proc/device-tree -o flattened.dts -
Generate a custom device-tree: You can configure Buildroot to compile custom device tree files (see BR2_LINUX_KERNEL_CUSTOM_DTS_PATH). This mechanism is used in this example to generate a functional device tree for the CM4 (the default bcm2711-rpi-cm4.dts results in an unusable system as the serial console UART is non-functional (conflicts with bluetooth) and USB is disabled. To use this method, you will likely need to migrate the various overlay files you need for your system into dtsi files you can use when building a flattened device tree. Refer to the included custom.dtsi (and the original source for the overlay files) for hints, read thorugh the Raspberry Pi Device Tree Overview, and diff your flattened tree with a known working run-time device tree loaded by the RPi firmware (see the dtc command above to get a dts file from your running system). To enable loading the device tree from your rootfs partition, edit cmdline.txt on the vfat partition and delete the fw_dft argument.
-
Load device tree overlays at run-time: You can use the kernel's configfs interface to load and unload device-tree overlays once the system is up and running by creating and removing directories and dtbo files in:
/sys/kernel/config/device-tree/overlays/
The default configuration ignores the devcie tree files on your rootfs partition and instead uses the device tree passed in by the RPi firmware. To use the flattened device tree file from your rootfs, edit cmdline.txt on the vfat partition and delete the fw_dft argument. This was done as it is expected more people would be confused by changes to config.txt having no effect vs. edits to custom.dtsi being ignored. I trust if you can edit dtsi files you can read instructions and/or trace the boot process and figure out what's going on. :-)
The Raspberry Pi boot process is managed by closed source firmware, and you may not be able to generate a working system without replicating some of the functionaltiy provided by these closed source applications. To debug use dtc to generate sorted dts files from your flattened dtb and the firmware generated run-time device tree which you can then compare using standard file diff tools.
You may still have to implement logic to deal with platform differences (eg: modify some device tree parameters based on SoC stepping revision).
The bootable sdcard.img image file is created by the genimage utility. Two filesystem images are created in addition to the rootfs image created by Buildroot:
- rootfs.ext4: Root filesystem image generated by Buildroot
- boot.vfat: 32M FAT filesystem with required RPi boot files and U-Boot
- data.ext4: 32M empty ext4 partition to use for persistent data (rauc.status)
The genimage.cfg file writes two copies of the boot filesystem to the disk image. The first is offset by 64K from the start of the device (to leave room for the U-Boot environment) and appears in the partition table. The second copy is offet 32M from the first and is not listed in the partition table (this is the "hidden" space used by RAUC with the boot-mbr-switch slot type).
Two copies of the root filesystem are then written to the disk, both with enries in the partition table. These are the "A" and "B" slots used by RAUC as the symmetric rootfs Slots.
Finally, a small 32M ext4 partition is created to store persistent data across updates. This partition is mounted with the flag "data=journal" to insure maximum safety. This partition is useful for storing occasionally modified data that must be retained across updates (eg: rauc.status file, static IP address, etc.). If you need to consistently write large amounts of data, you will likely want to change how persistent data is handled. See "Data Storage and Migration" in the RAUC manual.
RAUC requires updates to be cryptographically signed. This example includes a self-signed certificate expiring in 2033 for convenience. You must create a proper signed certificate prior to using this example for anything other than testing.
The RAUC configuration includes three slots that can be targeted for updates. Two rootfs slots are used in an "A B" redudnant setup (see "Symmetric Root-FS Slots" in the RAUC manual). The third slot is for the bootloader with a type of boot-mbr-switch, used for atomic bootloader updates. See the /etc/rauc/system.conf file for full details.
The RAUC update bundles are made by the post-image.sh script by creating a simple manifest file in a temporary directory along with the file system images created by the genimage tool. Since the bootloader is typically updated much less frequently than the root filesystem, two separate bundles are created. See the post-image.sh script for details.
- Example application that interacts with the systemd watchdog and triggers rauc status set-good once everything is up and running.
- Boot-time configuration of watchdog timer (RPi watchdog code was pulled from U-Boot in 2019).
As with almost all open source projects, this project would not be possible without the work of many others. I would like to extend my thanks and gratitude to:
- Buildroot, RAUC, and Das U-Boot communities: Obviously my work here would be impossible without the foundation proivded by these excellent projects. My contributions are a small addition to the great strides made by these folks.
- Bootlin: Other than the official documentation, the folks at Bootlin were probably my greatest resource. Their blog posts and online training materials were invaluable as I was trying to learn how all the pieces fit together.
- Home Assistant Operating System: This platform runs on a RPi4 (among others) and provided an example of a working Buildroot, U-Boot, and RAUC system and was very helpful as a reference.
- Raspberry Pi Foundation: In addition to making excellent, powerful, and affordable hardware, they provide excellent documentation for their systems.
As a Buildroot external directory tree which has borrowed heavily from the stock configurations and scripts proivded with Buildroot, this project is licensed under the same terms as the Buildroot project: GNU General Public License, version 2 or (at your option) any later version, with the exception that any package patches are covered by the the license of the software to which the patches are applied.