This arcticle discribes a way to run OpenShift as a workstation with GPU PCI passthrough and using Container Native Virtualization (CNV) in order to provide a virtualized Desktop experience running on a single OpenShift node.
This is used to run Microsoft FlightSimulator in a Windows VM with performances close from a Bare metal Windows installation.
In the future, a few improvements will be worked on: * Reducing the Control Plane footprint by relaying on microshift instead. * Using GPU from containers instead of virtual machines for Linux Desktop.
The workstation in use for this demo is a compact, mini-ITX workstation with the following hardware: * AMD Ryzen 9 3950X 16-Core 32-Threads * 64GB DDR4 * Nvidia RTX 3080 * 2x NVMe Disks * 1x SSD Disk * Single NIC 1GB
In order to avoid boot order conflicts, the OpenShift assisted installer will format the 512 first bytes of any disks that contains a bootable partition.
Therefore, in order to avoid loosing any existing system it is important to backup and remove any existing EFI/BIOS and boot partitions from already existing disks.
Once any existing file system is backed up and there is no more bootable partitions we can proceed with the OpenShift Single Node install.
It is important to note that CoreOS, the underlying operating system requires an entire disk for installation.
Here, we will keep the two NVMe disks for the persistant volumes as LVM Physical volumes belonging to a same Volume Group and we will use the SSD disk for the OpenShift operating system.
link:install/get-ocp-binaries.sh[role=include]link:install/install-config.yaml[role=include]mkdir ocp && cp install-config.yaml ocp
openshift-install --dir=ocp create single-node-ignition-configalias coreos-installer='podman run --privileged --rm \
-v /dev:/dev -v /run/udev:/run/udev -v $PWD:/data \
-w /data quay.io/coreos/coreos-installer:release'
cp ocp/bootstrap-in-place-for-live-iso.ign iso.ign
coreos-installer iso ignition embed -fi iso.ign rhcos-live.x86_64.iso
dd if=discovery_image_sno.iso of=/dev/usbkey status=progressOnce the ISO is copied to the USB drive, you can use the USB drive to boot your workstation node and install OpenShift Container Platform.
Activate Intel VT or AMD-V hardware virtualization extensions in BIOS.
link:install/cnv-resources.yaml[role=include]oc apply -f cnv-resources.yamlInstalling the Virtctl client on your desktop.
subscription-manager repos --enable cnv-4.10-for-rhel-8-x86_64-rpms
dnf install kubevirt-virtctlAs we do not need to manage LVM volumes automatically we would like to avoid automatically formating Logical Volumes once they are deleted from OpenShift.
While this could lead to data leak in a multi-tenant environment, removing the Local Storage Operator also avoid loosing your Virtual Machine partitions once you delete it.
As our GPU is the only one attached to the node a few additional steps are required.
We will use MachineConfig to configure our node accordingly.
All MachineConfig are applied on the master machineset because we have a single node OpenShift. With a multi nodes cluster those would be applied to worker instead.
To prevent Host system to bind console to GPU
https://www.reddit.com/r/VFIO/comments/cktnhv/bar_0_cant_reserve/ https://www.reddit.com/r/VFIO/comments/mx5td8/bar_3_cant_reserve_mem_0xc00000000xc1ffffff_64bit/
link:machineconfig/100-sno-kernelargs.yaml[role=include]|
Note
|
If you’re using an Intel CPU you’ll have to set intel_iommu=on instead. |
We deactivate the EFI Frambuffer with efifb:off
We first gather the PCI Vendor and product IDs from pciutils.
lspci -nn |grep VGAlink:machineconfig/100-sno-vfiopci.bu[role=include]dnf install butane
butane 100-sno-vfiopci.bu -o 100-sno-vfiopci.yaml
oc apply -f 100-sno-vfiopci.yamllink:machineconfig/98-sno-vtconsole-unbind.yaml[role=include]oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnvapiVersion: hco.kubevirt.io/v1
kind: HyperConverged
metadata:
name: kubevirt-hyperconverged
namespace: openshift-cnv
spec:
permittedHostDevices:
pciHostDevices:
- pciDeviceSelector: "10DE:2206"
resourceName: "nvidia.com/GEFORCE_RTX_3080"
- pciDeviceSelector: "10DE:1AEF"
resourceName: "nvidia.com/GEFORCE_RTX_3080_AUDIO"
...In order to directly connect a mouse, keyboard, audio device etc directly to the VM we passthrough one if the USB controller directly to the VM.
We first need to indentify it using pciutils.
lspci -nnkAfter selecting the USB Controller we want to dedicate to the Virtual Machine we should verify that this is the only PCI device in its IOMMU group. We first look for the PCI address in the iommu_groups folder structure, the list the PCI addresses belonging to this IOMMU group.
find /sys/kernel/iommu_groups/ -iname "*0b:00.3*"
ls /sys/kernel/iommu_groups/27/devices/Ref: https://access.redhat.com/solutions/6250271 Ref: https://kubevirt.io/user-guide/virtual_machines/host-devices/#listing-permitted-devices
Once identified we add its Vendor and product IDs to the list of permitted Host Devices.
Currently, Kubevirt does not allow providing a specific PCI address, therefore the pciDeviceSelector will match all similar USB Host Controller from the node. However, as we will only bind the one we are interested in to the VFIO-PCI driver the other ones will not be available for pci passthrough.
oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnvapiVersion: hco.kubevirt.io/v1
kind: HyperConverged
metadata:
name: kubevirt-hyperconverged
namespace: openshift-cnv
spec:
permittedHostDevices:
pciHostDevices:
- pciDeviceSelector: "1222:164F"
resourceName: "amd.com/XHCI_USB3_Controller"
...link:machineconfig/98-sno-xhci-unbind.yaml[role=include]|
Caution
|
double backslash with systemd in pci device id path https://www.freedesktop.org/software/systemd/man/systemd.syntax.html |
The virtual machine will use existing LVM Logical volumes, here we will assume we already have the Operating System installed on the LV with a UEFI boot.
Binding PV and PVC by label https://docs.openshift.com/container-platform/3.3/install_config/storage_examples/binding_pv_by_label.html
link:pv/fedora35.yaml[role=include]The virtual machines we will use as Desktops comes with a few specities: * We will passthrough the entire GPU Ref: https://kubevirt.io/2021/intel-vgpu-kubevirt.html * We will remove the existing default virtual VGA ref: https://kubevirt.io/api-reference/master/definitions.html#_v1_devices * We will passthrough en entire USB controller * We will use UEFI boot to be closer from typical BareMetal Ref: https://docs.openshift.com/container-platform/4.10/virt/virtual_machines/advanced_vm_management/virt-efi-mode-for-vms.html
link:vms/fedora35.yaml[role=include]This chapter is used as reference to the furture improvements to make.