Create a baseline build script for Ubuntu

Question

Create a baseline build script for Ubuntu

Closed this issue 5 years ago · 7 comments

Nomad desktop and the Nitrux OS is currently being built on the back of Ubuntu LTS. As of now however there is no easy way to build an Ubuntu ISO or derivatives with OverlayFS working with the proper persistence that Nitrux has.

With that in mind a script file to build Ubuntu LTS ISO with persistence is needed for those wanting to take advantage of anything outside of a live ISO. With the current Nitrux ISO tool there is a lot of extra components currently not needed to build just a vanilla Ubuntu ISO.

Answer 1 · 2019-04-06T12:33:18.000Z

Note: This a baseline, adjustments have to be made. If you have questions create a new issue.

Requeriments

Knowledge in Bash.
Experience with a CI system (in this case Travis).
Experience with APT and dpkg.

Besides the scripts found in this repository, you will need two other tools. The first one is runch, runch will allow you to run commands in a chroot environment. And the second one is mkiso, mkiso will be the tool that will make the bootable ISO.

Goal

The following will result in an unmodified Ubuntu LTS image with support for OverlayFS and updates through znx. The scripts described in this answer can be adapted to other Linux distribution families not based on Debian.

Notes

Bear in mind that these scripts contain lines that refer to the Travis CI system as that is what we have used.
Configuration of the Travis CI build system is outside the scope of this answer.

How do the scripts work?

The first one, build.sh is what the CI system will run to create the ISO file. And the second script bootstrap.sh is the script that will contain your customizations, this script is executed by runch as part of build.sh.

The customizations that are defined in bootstrap.sh are loaded from the folder /configs. For the sake of clarity, the scripts detailed here will use the same folder name.

Build script

First, set up the working directories for the build.

#! /bin/sh 

# -- Exit on errors.

set -e


# -- Prepare the directories for the build.

BUILD_DIR=$(mktemp -d)
ISO_DIR=$(mktemp -d)
OUTPUT_DIR=$(mktemp -d)

CONFIG_DIR=$PWD/configs

From here on now, adjustments have to be made to the script like the name of the ISO image.


# -- The name of the ISO image.

IMAGE=

Here you will specify the base to use, in this case, the latest LTS.

# -- Prepare the directory where the filesystem will be created.

wget -O base.tar.gz -q http://cdimage.ubuntu.com/ubuntu-base/releases/18.04/release/ubuntu-base-18.04.2-base-amd64.tar.gz
tar xf base.tar.gz -C $BUILD_DIR


# -- Populate $BUILD_DIR.

wget -qO /bin/runc https://raw.githubusercontent.com/Nitrux/runc/master/runch
chmod +x /bin/runch

The contents of the config folder needed for vanilla Ubuntu are grub.cfg, loopback.cfg, and persistence. Everything else in the folder can be discarded for this purpose (using ours as an example).


cp -r configs $BUILD_DIR/

runc $BUILD_DIR bootstrap.sh || true

rm -rf $BUILD_DIR/configs


# -- Copy the kernel and initramfs to $ISO_DIR.

mkdir -p $ISO_DIR/boot

cp $(echo $BUILD_DIR/vmlinuz* | tr ' ' '\n' | sort | tail -n 1) $ISO_DIR/boot/kernel
cp $(echo $BUILD_DIR/initrd* | tr ' ' '\n' | sort | tail -n 1) $ISO_DIR/boot/initramfs

This line was added explicitly for Travis CI, due to the amount of time that it takes to compress the filesystem, Travis would kill the job, and it would result in a build error.

# -- Compress the root filesystem.

(while :; do sleep 300; printf "."; done) &


mkdir -p $ISO_DIR/casper
mksquashfs $BUILD_DIR $ISO_DIR/casper/filesystem.squashfs -comp xz -no-progress -b 1M

This line is also specific for Travis CI. With this, we know what commit generates an ISO image.


# -- Write the commit hash that generated the image.

printf "${TRAVIS_COMMIT:0:7}" > $ISO_DIR/.git-commit

Here mkiso will generate the ISO image.

# -- Generate the ISO image.

wget -qO /bin/mkiso https://raw.githubusercontent.com/Nitrux/mkiso/7f171c70b0ee26872afc732fec94518223777f36/mkiso
chmod +x /bin/mkiso

You need to replace the GRUB theme that we use, you can extract the Debian package that has the Ubuntu theme, then change the GRUB theme accordingly.


git clone https://github.com/Nitrux/nitrux-grub-theme grub-theme

mkiso \
	-d $ISO_DIR \
	-V "NITRUX" \
	-g $CONFIG_DIR/grub.cfg \
	-g $CONFIG_DIR/loopback.cfg \
	-t grub-theme/nomad \
	-o $OUTPUT_DIR/$IMAGE

Finally, you need to add the update information that znx will use to update the ISO image using zsync. Here you will need to use a URL where znx can look for the zsync file.


# -- Embed the update information in the image.

UPDATE_URL=
printf "zsync|$UPDATE_URL" | dd of=$OUTPUT_DIR/$IMAGE bs=1 seek=33651 count=512 conv=notrunc


# -- Calculate the checksum.

sha256sum $OUTPUT_DIR/$IMAGE > $OUTPUT_DIR/$IMAGE.sha256sum


# -- Generate the zsync file.

zsyncmake \
	$OUTPUT_DIR/$IMAGE \
	-u ${UPDATE_URL/.zsync} \
	-o $OUTPUT_DIR/$IMAGE.zsync

In this last part, you tell the CI system where to upload it. The values will be taken from the setup of the Travis instance in this case.


# -- Upload the ISO image.

cd $OUTPUT_DIR

export SSHPASS=$DEPLOY_PASS

for f in *; do
    sshpass -e scp -q -o stricthostkeychecking=no $f $DEPLOY_USER@$DEPLOY_HOST:$DEPLOY_PATH
done

Bootstrap

In bootstrap.sh is where you will define the customizations that will be done to the built system. For this issue, we'll only mention what is essential for a vanilla Ubuntu ISO.

Here you will define the locale to be used.

#! /bin/bash


export LANG=C
export LC_ALL=C

Here you can define the list of packages that will be installed in the chroot.

# -- Packages to install.

PACKAGES='
dhcpcd5
user-setup
localechooser-data
cifs-utils
casper
lupin-casper
xz-utils
ubuntu-desktop

This part will install the necessary packages for the CI system to continue with the build process.

# -- Install basic packages.

apt -qq update > /dev/null
apt -yy -qq install apt-transport-https wget ca-certificates gnupg2 apt-utils --no-install-recommends > /dev/null

You can use a customized sources.list file, this file is needed for APT to download the packages.

# -- Use optimized sources.list.

cp /configs/sources.list /etc/apt/sources.list

In this is the part the script will install the packages from the previous section. You can choose to install the recommended packages or not. Using --no-install-recommends more or less equates to using the new minimal installation option in Ubiquity.

# -- Update packages list and install packages, avoiding recommended packages.

apt -qq update > /dev/null
apt -yy -qq upgrade > /dev/null
apt -yy install ${PACKAGES//\\n/ } --no-install-recommends

In this part of the script is where the magic happens, the file persistence is what defines what directories in the filesystem hierarchy will work with OverlayFS (check Configs).

# -- Update the initramfs.

cat /configs/persistence >> /usr/share/initramfs-tools/scripts/casper-bottom/05mountpoints_lupin
update-initramfs -u

Finally, the script removes the additional packages like casper.

# -- Clean the filesystem.

apt -yy -qq purge --remove casper lupin-casper > /dev/null
apt -yy -qq autoremove > /dev/null
apt -yy -qq clean > /dev/null

Configs

In this directory is where you will add the configuration files that you will be using for your ISO. Of the contents of this directory you will be using:

10-globally-managed-devices.conf (solves a bug)
grub.cfg and loopback.cfg
persistence
sources.list (You can generate a new file using http://repogen.simplylinux.ch)

The contents of the file persistence are what will define what directories will use OverlayFS. In our case, we only enable persistence on the /etc and /home folders.

#! /bin/sh

for o in $(cat /proc/cmdline); do

	case $o in

		# -- Determine the overlay mountpoints from /proc/cmdline.

		ZNX_OVERLAYS=*)

			ZNX_OVERLAYS=${o#ZNX_OVERLAYS=}
			ZNX_OVERLAYS=$(printf $ZNX_OVERLAYS | sed 's/,/ /g')

		;;

		# -- Find the persistent storage directory.

		iso-scan/filename=*)

			PERSISTENT_DATA=${o#iso-scan/filename=}
			PERSISTENT_DATA=${PERSISTENT_DATA%/*}/DATA

		;;

	esac

done


# -- Mount the requested overlays.

for k in $ZNX_OVERLAYS; do

	mkdir -p \
		/root/$k \
		/root/isodevice/.overlay-tmp/$k.w \
		/root/isodevice/$PERSISTENT_DATA/$k \
		/root/isodevice/.overlay-tmp/$k.w

	mount -t overlay \
		-o lowerdir=/root/$k \
		-o upperdir=/root/isodevice/$PERSISTENT_DATA/$k \
		-o workdir=/root/isodevice/.overlay-tmp/$k.w \
		. /root/$k

done

The configuration file of GRUB should look something like this:

set gfxmode=auto
insmod efi_gop
insmod efi_uga
insmod gfxterm
terminal_output gfxterm
loadfont /boot/grub/themes/ubuntu/dejavu_sans_mono_bold.pf2

set theme=/boot/grub/themes/ubuntu/theme.txt

menuentry "Boot Ubuntu " {
	set gfxpayload=keep
	linux /boot/kernel boot=casper quiet splash elevator=noop iso-scan/filename=$iso_path username=ubuntu hostname=ubuntu ZNX_OVERLAYS=/
	initrd /boot/initramfs
}

menuentry "Boot Ubuntu debug mode" {
	set gfxpayload=keep
	linux /boot/kernel boot=casper xforcevesa elevator=noop iso-scan/filename=$iso_path username=ubuntu hostname=ubuntu ZNX_OVERLAYS=/
	initrd /boot/initramfs
}

Note that the line ZNX_OVERLAYS=/ is added as a parameter. We have modified znx to make it easier to define what directories in the filesystem will use the OverlayFS driver. In this example, setting the parameter to ZNX_OVERLAYS=/ will use the overlay driver for the entire root filesystem.

The file loopback.cfg should point to grub.cfg

source /boot/grub/grub.cfg

Answer 2 · 2019-04-07T15:12:42.000Z

I tried to build the Nitrux ISO as I'm not familiar with Travis CLI and got a failed build:

Is there a GUI method or any terminal method of adding the OverlayFS without using Travis CLI? Or any help on where my build failed would be helpful as well.

Answer 3 · 2019-04-09T00:08:31.000Z

@linuxhelmet Please note that my response isn't complete yet.

Answer 4 · 2019-04-09T04:50:09.000Z

Sorry about that. Also what I tried to build above and failed was the Nitrux ISO not an Ubuntu ISO. I wanted to see if I could recreate the Nitrux ISO using your repositories before building a vanilla Ubuntu ISO using these new scripts but failed. I will let you finish responding I know these things take time just showing you what I've tried in the meantime.

Answer 5 · 2019-05-05T22:55:10.000Z

The response above should serve as a baseline to create a vanilla Ubuntu 64-bit, and UEFI only ISO file with persistence enabled and support for the features of znx.

Further changes are up to the user, and as indicated, the configuration of Travis CI is outside of the scope of this issue.

Answer 6 · 2019-05-31T07:06:23.000Z

Sorry about that. Also what I tried to build above and failed was the Nitrux ISO not an Ubuntu ISO. I wanted to see if I could recreate the Nitrux ISO using your repositories before building a vanilla Ubuntu ISO using these new scripts but failed. I will let you finish responding I know these things take time just showing you what I've tried in the meantime.

@linuxhelmet Your build fails because you are using our same configuration for your Travis instance, you must change that accordingly. At the end of the log, Travis notifies you that you are missing files, specifically bootstrap.sh.

I successfully created a build of elementary that utilizes znx. You can check how I built it in the branch elementary-znx and use it as a guide to building your Ubuntu image.

Answer 7 · 2020-04-20T10:39:42.000Z

I created a branch that builds a vanilla Ubuntu image ubuntu-znx since the guide above is outdated in various aspects; however, it can be a reference.

The scripts now have comments in them and are quite easy to understand.