/libvirt_win10_vm

Primary LanguageShell

Virtual Machine with vGPU Unlock for single GPU desktop

This document catalogs my working setup for configuring an Arch Linux host OS to run a Windows 10 VM while passing through a virtualized GPU.

Sources

The resources I used during my own install. This document is a compilation of the instructions and files provided by these people.

Guides

Sources

Configuration

The what and how. Mostly described in suggested installation order unless otherwise stated.

Host

Operating system

Configuration of the host OS for virtualization using KVM and device passthrough using OVMF, IOMMU, and VFIO.

Arch Linux
Kernel 5.12.15-arch1-1
Kernel modules
  • vfio
  • vfio_iommu_type1
  • vfio_pci
  • vfio_virqfd
How to add
  1. Edit /etc/mkinitcpio.conf
  2. Add MODULES=(... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ...) to the MODULES line.
  3. Add HOOKS=(... modconf ...) to the HOOKS line.
  4. Update update-initramfs -P (use mkinitpico -P for manjaro)
Boot parameters
  • amd_iommmu=on
  • iommu=pt
  • vfio_iommu_type1.allow_unsafe_interrupts=1
  • kvm.ignore_msrs=1
  • blacklist=nouveau
How to add
  1. edit /etc/default/grub
  2. Add to GRUB_CMDLINE_LINUX_DEFAULT:
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 blacklist=nouveau ..."
  1. Update GRUB update-grub

Hardware

The physical devices present on my system.

  • CPU: AMD Ryzen 7 5800X (16) @ 3.800GHz
  • GPU: NVIDIA GeForce GTX 1060 6GB
  • Memory: 16MiB
  • 1440p Monitor

Software

What needs to be installed.

Arch Packages
Packages installed using pacman.
linux

The OS kernel.

Version: 5.12.15How to installFirst option
  1. Install downgrade from AUR (using yay or other AUR helper)
  2. downgrade linux
  3. Select 5.12.15
From Arch archive
  1. Download package curl -O https://archive.archlinux.org/packages/l/linux/linux-5.12.15.arch1-1-x86_64.pkg.tar.zst
  2. Install with pacman -U linux-5.12.15.arch1-1-x86_64.pkg.tar.zst
linux-headers

Needed to build kernel modules, particularly the Nvidia driver.

Version: 5.12.15How to installFirst option
  1. Install downgrade from AUR (using yay or other AUR helper)
  2. downgrade linux-headers
  3. Select 5.12.15
From Arch archive
  1. Download package curl -O https://archive.archlinux.org/packages/l/linux-headers/linux-headers-5.12.15.arch1-1-x86_64.pkg.tar.zst
  2. Install with pacman -U linux-headers-5.12.15.arch1-1-x86_64.pkg.tar.zst
virt-manager

Used to create and run virtual machines.

libvirt
kvm

(Kernel-based Virtual Machine). A hypervisor. Used for virtualization.

qemu
looking-glass

Used to view VM in graphical display.

iptables-nft

For networking with the VM.

edk2-ovmf
dnsmasq

Used for VM networking.

mdevctl

Used to create vGPU devices

scream

Used to read audio stream from network.

How to setup
  1. Install using package manager.
  2. Create systemd-file 
    [Unit]
Description=Scream network audio client

[Unit]
Description=Scream Receiver
After=pulseaudio.service network-online.target
Wants=pulseaudio.service

[Service]
Type=simple
ExecStartPre=/bin/sleep 3
ExecStart=scream -i virbr0
Restart=always
RestartSec=1

[Install]
WantedBy=default.target
Python packages
Installed with pip.
frida
How to installIf using conda
  1. Install python-pip using pacman.
  2. Run /bin/pip install frida
Otherwise
  1. pip install frida
Graphics setup
Nvidia Driver
Version: 460.73.01 merged and pre-unlocked This driver is a merged version of Nvidia’s vGPU driver and their standard driver. It allows for the host graphics to be displayed while the vGPU’s are active.Components
  1. merged driver
  2. vgpu_unlock
  3. vgpu_unlock_5.12 patch
How to installMake sure all above arch packages and python packages are installed.Download thecomponents using curl -O the_linkMake the driver executable. chmod +x NVIDIA-Linux-x86_64-640.73.01-grid-vgpu-kvm-v5.runExtract it. ./NVIDIA-Linux-x86_64-640.73.01-grid-vgpu-kvm-v5.run -xcd NVIDIA-Linux-x86_64-640.73.01-grid-vgpu-kvm-v5Apply the 5.12 patch patch -p0 < ../twelve.patchDisable graphics sudo systemctl isolate multi-user.targetRe-login as root and navigate back to the driver directory 
cd /home/user/Downloads/NVIDIA-Linux-x86_64-460.73.01-grid-vgpu-kvm-v5
Install the driver 
./nvidia-installer --dkms
Add vgpu_unlock to service files. 
nano /lib/systemd/system/nvidia-vgpud.service

Replace ExecStart with:

ExecStart=/root/vgpu_unlock/vgpu_unlock /usr/bin/nvidia-vgpud
  
nano /lib/systemd/system/nvidia-vgpu-mgr.service

Replace ExecStart with:

ExecStart=/root/vgpu_unlock/vgpu_unlock /usr/bin/nvidia-vgpu-mgr
  
systemctl daemon-reload
Add vgpu_unlock to source files. 
nano /usr/src/nvidia-460.73.01/nvidia/os-interface.c

Under #include “nv-time.h” insert this line

#include "/root/vgpu_unlock/vgpu_unlock_hooks.c"
  
nano /usr/src/nvidia-460.73.01/nvidia/nvidia.Kbuild

Add this to the bottom of the file.

ldflags-y += -T /root/vgpu_unlock/kern.ld
Remove and reinstall the Nvidia dkms module 
dkms remove -m nvidia -v 460.73.01 --all
dkms install -m nvidia -v 460.73.01
Make all of vgpu_unlock executable

chmod -R +x vgpu_unlock

Reboot
vGPU Profiles
  • nvidia-48 GRID P40-3Q
How to installChoose profile 
mdevctl types

I chose nvidia-47 (GRID P40-3Q) because it has 3Q at the end of its name. This means I can have two 3GB vGPU’s running.

Determine GPU PCI address 
lspci | grep VGA
PCI_ADDRESS=$(lspci | grep VGA | grep -Po '\d\w:\d\d.\d')
echo $PCI_ADDRESS
Generate UUIDs for each vGPU 
UUID1=$(uuidgen)
UUID2=$(uuidgen)
echo $UUID1 $UUID2
Create vGPUs 
mdevctl start -u $UUID1 -p 0000:$PCI_ADDRESS -t nvidia-48
mdevctl start -u $UUID2 -p 0000:$PCI_ADDRESS -t nvidia-48

mdevctl define -a -u $UUID1
mdevctl define -a -u $UUID2

VM Image

The virtual machine configuration.

  • Chipset: Q35
  • Firmware: UEFI x86_64: /usr/share/edk2-ovmf/x64/OVMF_CODE.fd
  • Hypervisor KVM
  • Spice Server
  • Video QXL
  • MDEV
  • shmem looking-glass
How to install
In virt-manager, create virtual machine.
Apply above settings minus “MDEV” and “shmem looking-glass”.
Start VM and install OS.
Shutdown guest.
Add looking-glass shmem device.

The size is determined like this.

<shmem name='looking-glass'>
  <model type='ivshmem-plain'/>
  <size unit='M'>64</size>
</shmem>

Make a file /etc/tmpfiles.d/10-looking-glass.conf

#Type Path               Mode UID  GID Age Argument

f /dev/shm/looking-glass 0660 user kvm -
Remove tablet device
Add MDEV device.

Where $UUID1 is the UUID of one of the mdev vGPU’s created earlier.

<hostdev mode="subsystem" type="mdev" managed="no" model="vfio-pci" display="on">
  <source>
    <address uuid="$UUID1"/>
  </source>
</hostdev>
Install all guest software.
Shutdown guest.
Either set Video model to “none” or install guest Nvidia driver​.
How to run
  1. Start VM. (May need to click run several times if error related to “vfio-<bus>” occurs.
  2. Run looking-glass client 
    export SDL_VIDEO_X11_VISUALID=
looking-glass-client

Guest

Inside the VM.

Operating system

  • Windows 10

To download (within guest)

Software

Looking Glass Host
How to install
  1. Download components.
  2. Update driver in Device Manager->System Devicess->PCI standard RAM Controller to virtio PCI RAM controller driver.
  3. Install virtio guest tools and spice guest tools.
  4. Install looking glass host by running looking-glass-host-setup.exe as administrator.
  5. VM.
Nvidia GRID driver
How to install
  1. Run Nvidia GRID driver installer.
  2. Reboot.
  3. In Nvidia Console set resolution. My monitor used 2560x1440.
  4. Run vGPU_LicenseBypass.
Scream audio driver
How to install
  1. Install driver.
  2. Change sound source to Scream (WDM).

My files

Suggestions

Since there is a lot of messing with the kernel and graphics drivers, it’s important to have a plan for recovering your host OS.

  • Use timeshift or other auto-snapshotting of the host.
  • Have a recovery drive available.

Maintenance

Preventing accidental upgrades

In order to avoid accidentally upgrading these carefully chosen package versions, make sure to ignore updates to them:

IgnorePkg = linux linux-headers nvidia-dkms nvidia nvidia-utils nvidia-settings opencl-nvidia libxnvctrl