transactional-netboot

Set up the server

Use any working Linux system with NFS and tftp servers. openSUSE Leap 15.0 and Tumbleweed are known to work.
Install transactional-netboot from obs://home:favogt:nfsroot/
Create a directory containing all minion systems, for instance /srv/minion. The directory needs to be on btrfs, with CoW enabled, but not part of snapper's snapshots. Create a ".install" subdirectory.
Edit /etc/sysconfig/tftp, set TFTP_DIRECTORY="/srv/minion".
As long as the whole /srv/minion tree is accessible from NFS and tftp under the same path, you can choose to export a different path as well. This means you can also export parent directories or use bind mounts.
Adjust the values in /etc/transactional-netboot.conf as necessary.
Edit /etc/exports, export the directory as noted. Make sure that it is readable by all clients. Example:
/srv/minion 192.168.42.0/24(ro,async,no_subtree_check,no_root_squash,fsid=0). The async,fsid=0 parameters are not necessary.
Export the .install subdirectory as rw,nohide,crossmnt. Example:
/srv/minion/.install 192.168.42.0/24(rw,async,no_subtree_check,no_root_squash,nohide,crossmnt)

Set up a minion

In this example, openSUSE Tumbleweed will be installed as "tumbleweed" minion.

Make sure that the DHCP server used for netbooting gives out IPs to the nodes.

Run transactional-netboot tumbleweed --init. It will give you a target path for the installation.
Boot one of the target systems with the Tumbleweed installation media.
Proceed with the installation until the drive selection as usual. Ignore any warnings that no disks where discovered.
In the partitioning proposal, open the expert partitioner.
Add a new NFS mount with the local mountpoint / and as source hostname of the NFS server and the path transactional-netboot gave you.
Continue as usual, ignore any warnings related to booting or bootloader installation (boo#1090056)
After the installation is complete, run transactional-netboot tumbleweed --setup. If the install mountpoint is still busy, you might need to wait some time before trying again.
Include the file for dhcpd configuration given by transactional-netboot in your dhcpd configuration. If the DHCP server is running on a different server, copy it over. Set the next-server to the IP of the NFS/TFTP server.
Call transactional-netboot tumbleweed to open a shell in a new snapshot. In this shell you need to perform the initial configuration, explained in the next steps.
Edit /etc/sysconfig/bootloader, set LOADER_TYPE="none".
Edit /etc/default/grub, set
GRUB_CMDLINE_LINUX_DEFAULT="rd.neednet=1 ip=dhcp" GRUB_DISABLE_OS_PROBER="true"
SUSE_NFS_SNAPSHOT_BOOTING="true"
GRUB_DEVICE_BOOT="nfs"
GRUB_FS="nfs"
Write use_fstab="yes" into /etc/dracut.conf.d/42-nfsroot.conf
Follow either the guide for minions with volatile or persistent storage
Run mkinitrd
Leave the minion chroot with exit

Now you can boot the nodes using PXE.

Minion with volatile storage

A minion set up this way stores all changes in tmpfs.

Install read-only-root-fs-volatile, dracut and grub2 from obs://home:favogt:nfsroot
Install kernel-default from obs://home:favogt:overlay/standard

Advanced configuration: /home on overlay

Go into the minion chroot: transactional-netboot tumbleweed
Create /etc/systemd/system/tmp-overlay@.service with this content:

[Unit]
Description=Directories for overlay mounting of %I
Requires=tmp.mount
After=tmp.mount

[Service]
Type=simple
ExecStart=/usr/bin/mkdir -p /tmp/%I-upper /tmp/%I-work

Append a mountpoint for /home to /etc/fstab:
overlay /home overlay defaults,upperdir=/tmp/home-upper,workdir=/tmp/home-work,lowerdir=/home,x-systemd.requires=tmp-overlay@home.service 0 0
Exit the chroot: exit

You can use the same mechanism for e.g. /root by adding a line in /etc/fstab.

Minion with persistent storage

A minion with persistent storage has /var mounted from a local device.

Edit /etc/fstab, add entries for the minion's /var and /tmp:

/dev/vda1 /var ext4 defaults 0 0
tmpfs /tmp tmpfs defaults,nosuid,nodev 0 0

Install transactional-netboot-persistent and read-only-root-fs
Create /etc/transactional-netboot/create-varpart.sh (executable) which creates and copies over /var contents. Example:

#!/bin/bash

set -euo pipefail

PARTITION=/dev/vda1

if [ -e $PARTITION ]; then
        echo "Partition already there."
        return 0
fi

DISK=/dev/vda

cat <<EOF | fdisk $DISK
g
n
1


p
w
EOF

partprobe $DISK
mkfs.ext4 $PARTITION
mount $PARTITION /mnt
rsync -aAXH /var/ /mnt/
sync
umount /mnt

Minion with /home on a persistent overlay

Go into the minion chroot: transactional-netboot tumbleweed
Create /etc/systemd/system/var-overlay@.service with this content:

[Unit]
Description=Directories for overlay mounting of %I
Requires=var.mount
After=var.mount

[Service]
Type=simple
ExecStart=/usr/bin/mkdir -p /var/lib/overlay/%I /var/lib/overlay/work-%I

Append a mountpoint for /home to /etc/fstab:
overlay /home overlay defaults,upperdir=/var/lib/overlay/home,workdir=/var/lib/overlay/work-home,lowerdir=/home,x-systemd.requires=var-overlay@home.service 0 0
Exit the chroot: exit