/kvm-vfio-notes

Notes on getting KVM VFIO working on my hardware

Primary LanguageShell

kvm-vfio-notes

Notes on getting KVM VFIO working on my hardware:

As of 2024-12-05, KVM VFIO passthrough of a current-gen Nvidia card while retaining the Ryzen 5700G's IGP as the host's framebuffer console is fully working.

I now provide a set of screenshots documenting every BIOS option, whether I changed it or not, as a means of providing a "known-good snapshot" of a working BIOS configuration. You will find them in the /screenshots directory of this repo.

[ January 2025 notes: ]
19-Jan-2025 I had a shitty USB device installed. Once removed, BIOS B55AK830 works as expected. I apologize for slandering a perfectly fine BIOS.

16-Jan-2025
I feel like the system is flakier about POSTing with BIOS B55AK830. In particular, it hangs around on BIOS POST code B4 (USB init) for SEVERAL MINUTES each reboot before moving normally on through the rest of the POST process. I have made no effort thus far to determine of this this is an unrelated post-hoc red herring, or a regression associated with the new BIOS version.

While exploring the idea of switching to a card with more than my current 20GB VRAM, attempted to set "Above 4G decoding" to "Enabled" and thereafter the dependent suboption "Resizable BAR" to "Auto," but found that while the system POSTs, continues to direct its video output correctly through the iGPU, and boots the OS, the combination of these two options being enabled causes qemu/KVM to fail with an "internal error" when attempting to start a VM with the RTX 4000 SFF Ada Generation 20GB card passed through to it.
With only "Above 4G decoding" enabled but "Resizable BAR" disabled, VMs start, run, and access the passed-through GPU normally.
VMs also work normally and are able to access all of the available VRAM address space without these options enabled.
MY GUESS: this setting only matters when attempting to use the card directly on the host system and through its UEFI, rather than within the VM, which implements its own configuration. My configuration tries to get the host system to entirely ignore the discrete GPU, by not initializing it in the UEFI nor when Linux boots, leaving it unsullied and fully available to be taken over by a VM.

[ December 2024 notes: ]

  1. I have found that the linux-amd-znver3 kernel reliaby works for passthrough, initializing devices in a sane order and respecting boot-time blacklist kernel args https://aur.archlinux.org/packages/linux-amd-znver3.
  2. I have now fully migrated over to openSUSE, and the stock kernel there also works perfectly. However, since I am chasing optimization, I have created a synthesis of the Arch linux-amd-znver3 and openSUSE stock kernel configs, which I maintain separately here: https://github.com/theodric/linux-amd-zen1-zen2-zen3-openSUSE_TW
  3. I have since managed to get the Nvidia RTX A2000 12GB working in passthrough
  4. The RTX 4000 SFF Ada Generation 20GB works as a drop-in replacement for the A2000.
  5. Stability is much improved - I don't know what to blame, but crashes are infrequent enough (i.e. one crash every 6-8 months) that I haven't spent time on it.
  6. I have recently transitioned away from Arch to openSUSE Tumbleweed, which works out-of-the-box as a passthrough host and does not constantly break. [ December 2024 note: openSUSE is love, openSUSE is life. ]
  7. I am separately maintaning a variant of the linux-amd-znver3 and linux-amd-znver2 configs which have been tweaked to work on openSUSE and are available at https://github.com/theodric/linux-amd-zen2-zen3/

2022 NOTES - FULLY WORKING ALL-AMD PASSTHROUGH

In this configuration, I have two fully independent "computer stations" that can be used simultaneously:

  1. the host Linux system, on which I have a KDE desktop
  2. the guest VM, which can be Windows, Linux, or (theoretically) macOS (but that's illegal, so don't break the law, OK?)

Arch Linux CPU: Ryzen 7 5700G MB: Biostar B550T-SILVER Host GFX: Ryzen 7 5700G iGPU Passthrough GFX: PCIe MSI Radeon 560 ITX (fully working in Windows and Linux guests)

In UEFI setup:

In Advanced -> PCI Subsystem Settings:

  • Enable Above 4G decoding
  • Disable Resizable BAR
  • Enable SR-IOV support
  • Enable BME DMA Mitigation

In Advanced -> CSM Support

  • Option ROM Execution:
  • Network - UEFI
  • Storage - UEFI
  • Video - Legacy - forces PCIe GPU to legacy mode so it doesn't get grabbed by UEFI, which would stop it from being free for passthrough to the VM
  • Other PCI Device - UEFI

In Chipset:

  • In North Bridge Configuration
  • IOMMU -> Enabled
  • Primary video device -> IGD Video

append kernel args: iommu=pt and vfio-pci.ids=

Find those IDs and check their PCIe group assignments with this script:

[root@cube ~]# cat vfio.sh
#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

I am passing through 1 or 2 ports of the onboard USB 3.2 controller that is in its own PCIe device group, because the other controllers are in groups with other devices and don't break out cleanly even with the additional kernel args, resulting in instability.

The WLAN socket on this board functions perfectly as a standard PCIe 4x slot with an adapter cable, but again, it's part of a group of several other devices. However, adding a USB3 card here helps make up for the ports lost to passthrough.

-> TODO: see if my PCIe switch works on this

Works with all kernels tested so far:

[root@cube ~]# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-linux-zen root=UUID=cbde20e5-0804-4179-ae95-7c1752d86f04 rw loglevel=3 net.ifnames=0 biosdevname=0 noibrs noibpb nopti nospectre_v2 nospectre_v1 l1tf=off nospec_store_bypass_disable no_stf_barrier mds=off tsx=on tsx_async_abort=off mitigations=off iommu=pt vfio-pci.ids=1002:67ff,1002:aae0,1022:1639

After that, it's just a matter of assigning the PCIe devices and/or host USB devices to the VM in either the virt-manager GUI, or else directly in the config file if you're a masochist.

To get rid of the Spice display that is auto-populated, you may have to resort to editing XML, because the GUI will often refuse to let you delete it, citing some dependency.

After that, you can just toggle the virtual (non-GPU) display on and off as required by changing the Video -> Model setting in the VM properties between "None" and the other options.

######### OLD 2021 NOTES - PROXMOX ON MAC PRO 5,1 ################

The device to be passed through can't be held by any driver.

OSX-KVM instructions say to blacklist the graphics card driver and use only integrated graphics, but that doesn't work for me because my Socket 1366 Xeon does not have onboard graphics, and both of the video cards in the system use the amdgpu kernel module.

The solution is to use the sysfs interface to unbind the device I will pass through to macOS from the driver, like so:

root@kvm-mule:/sys/bus/pci/drivers/amdgpu# lspci -vv | grep -i amd
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 550 640SP / RX 560/560X] (rev cf) (prog-if 00 [VGA controller])
    Kernel driver in use: amdgpu
    Kernel modules: amdgpu
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X]
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] (prog-if 00 [VGA controller])
    Kernel modules: radeon, amdgpu
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti HDMI Audio [Radeon HD 7870 XT / 7950/7970]

root@kvm-mule:/sys/bus/pci/drivers/amdgpu# echo -n 0000:01:00.0 > unbind #detach - works live
root@kvm-mule:/sys/bus/pci/drivers/amdgpu# echo -n 0000:01:00.0 > bind #reattach - works live