SNO seed image creation and restoration using Image Base Upgrade (IBU)

Note that this repo is just a proof-of-concept. This repo is for debugging / experimenting with single node OpenShift **

Note that single node OpenShift relocation is currently unsupported.

TL;DR Full run with vDU profile

Define a few environment variables:

SEED_IMAGE=quay.io/whatever/ostbackup:seed
export PULL_SECRET=$(jq -c . /path/to/my/pull-secret.json)
export BACKUP_SECRET=$(jq -c . /path/to/my/repo/credentials.json)

Create seed VM and seed image

make seed-vm-create wait-for-seed vdu seed-cluster-prepare seed-image-create SEED_IMAGE=$SEED_IMAGE

Create recipient VM and restore seed image

make recipient-vm-create wait-for-recipient recipient-cluster-prepare seed-image-restore SEED_IMAGE=$SEED_IMAGE
virsh reboot recipient

Prerequisites

NMState v2.2.10 or above, this is required due to the nmstate config used in the agent-config.yaml

sudo dnf copr enable nmstate/nmstate-git
sudo dnf install nmstate
- Set `PULL_SECRET` environment variable to the contents of your cluster pull secret
- Set `BACKUP_SECRET` environment variable with the credentials needed to push/pull the seed image, in standard pull-secret format

virt-install

sudo dnf install virt-install

Fedora prerequisites

If using systemd-resolved as a system DNS resolver

Show more info

Add the NetworkManager dnsmasq instance as a DNS server for resolved:

sudo mkdir /etc/systemd/resolved.conf.d

Then create /etc/systemd/resolved.conf.d/dns_servers.conf with:

[Resolve]
DNS=127.0.0.1
Domains=~.

And finally restart systemd-resolved:

sudo systemctl restart systemd-resolved

If using authselect as nsswitch manager

Show more info

Install libvirt-nss

sudo dnf install libvirt-nss

Add authselect libvirt feature

sudo authselect enable-feature with-libvirt

This makes it so that libvirt guest names resolve to IP addresses

How this works

Generate the seed image template

To generate a seed image we want to:

Provision a VM and install SNO in it

make seed-vm-create wait-for-seed

(OPTIONAL) Modify that installation to suit the use-case that we want to have in the seed image. In this example we install the components of a vDU profile

make vdu

Prepare the seed cluster to have a couple of needed extras

make seed-cluster-prepare

Create a seed image from that SNO cluster

make seed-image-create SEED_IMAGE=quay.io/whatever/repo:tag

This will run ibu-imager to create an OCI seed image

WARNING Once we create a seed-image, some changes will be applied to seed VM that will "break" the node. If you intend on reusing that seed VM, it would be wise to create a backup using make seed-vm-backup that you can restore afterwards

Restore seed image

To restore a seed image we will use LifeCycle Agent, and manage everything with the CR ImageBasedUpgrade

The steps will be as follow:

Provision a VM and install SNO in it

make recipient-vm-create wait-for-recipient

Prepare the recipient cluster for a couple of extras (LCA operator, shared /var/lib/containers)

make recipient-cluster-prepare

Restore the seed image

make seed-image-restore SEED_IMAGE=quay.io/whatever/repo:tag

Reboot into the new deployment

virsh reboot recipient

Extra goodies

Descriptive help for the Makefile

You can run

make help

and get a description of the main Makefile targets that you can use

Backup and reuse VM qcow2 files

To be able to reuse the VMs, we can backup the qcow2 files of both seed and recipient VM This will allow us to skip the initial provision, allowing for faster iterations when testing To create a backup run:

make seed-vm-backup

make recipient-vm-backup

To restore an image, we run the complementary restore command

make seed-vm-restore

make recipient-vm-restore

IMPORTANT: Remember that certificates expire, so if a backed up image is old, certificates will expire and openshift wont be usable TODO: Apply recert after restoring the image

vDU profile

A vDU profile can be applied to the image before baking with

make vdu

Use shared directorty for /var/lib/containers

A shared directory /sysroot/containers can be used to mount and share /var/lib/containers among ostree deployments Run:

make seed-varlibcontainers

make recipient-varlibcontainers

This will create a /sysroot/containers in the SNO (when not specifying the cluster with the CLUSTER variable, it defaults to the seed image) to be mounted in /var/lib/containers The use case for this is to easily precache all the images that the cluster in the seed image will need, while original recipient cluster is still running

WARNING

It is important to note that for precaching to work, this change must be applied both in seed image and recipient cluster

Examples

Installing the backup into some running SNO

make seed-image-restore SNO_KUBECONFIG=path/to/recipient/sno/kubeconfig SEED_IMAGE=$SEED_IMAGE

Reboot the recipient host

mresvanis/ib-orchestrate-vm