Fedora 38 + PCI passthrough

Prerequisites

Fedora 38 (fresh install off the live USB image)
Computer with
- Two graphics cards
- Motherboard with the Intel 200 series chipset (Union Point) or newer
A fresh backup of anything you don't care to lose

ℹ️ The two graphics cards can be different models. If your cards are identical, you'll need to do some extra steps that are prefaced with (ACS only). These steps include compiling your own kernel to include Alex Williamson's patch to allow any PCIe device to use Access Control Services. More information on this patch, why it's necessary, and what it does available here. If your cards aren't identical, skip these steps.

Warning Having problems since kernel 5.14.x? Check out the notes!

🆕 Just want the RPMs? Go to the latest workflow run and download the RPMs as a build artifact, as shown below:

Setup and configuration of the host machine

Add RPM Fusion

sudo dnf install https://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-$(rpm -E %fedora).noarch.rpm https://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$(rpm -E %fedora).noarch.rpm

(ACS only) - Install the dependencies to start building your own kernel.

sudo dnf install fedpkg fedora-packager rpmdevtools ncurses-devel pesign bpftool

(ACS only) - Set up your home build directory (if you haven't ever built any RPMs before)
```
rpmdev-setuptree
```

(ACS only) - Install the kernel source and finish installing dependencies.

koji download-build --arch=src kernel-6.2.15-300.fc38.src.rpm
rpm -Uvh kernel-6.2.15-300.fc38.src.rpm
cd rpmbuild/SPECS/
sudo dnf builddep kernel.spec

(ACS only) - Add the ACS patch (link) as ~/rpmbuild/SOURCES/add-acs-override.patch.

curl -o ~/rpmbuild/SOURCES/add-acs-override.patch https://raw.githubusercontent.com/some-natalie/fedora-acs-override/main/acs/add-acs-override.patch

(ACS only) - Edit ~/rpmbuild/SPECS/kernel.spec to set the build ID and add the patch. Since each release of the spec file could change, it's not much help giving line numbers, but both of these should be near the top of the file. To set the build id, add the two lines near the top of the spec file with the other release information.
```
# Set buildid
%define buildid .acs
```
To add the patch, add the two lines below to the spec file in the section for patches (usually right below the sources).
```
# ACS override patch
Patch1000: add-acs-override.patch
```
Then tell it to apply the patch in the prep section. It will be below the ApplyOptionalPatch() function definition, normally right above the # END OF PATCH APPLICATIONS comment.
```
ApplyOptionalPatch add-acs-override.patch
```
(ACS only) - Compile! This takes a long time.
```
rpmbuild -bb kernel.spec
```
(ACS only) - Install the new packages!
```
cd ~/rpmbuild/RPMS/x86_64
sudo dnf localinstall *.rpm
```
ℹ️ You should now have at least the following packages installed: kernel, kernel-core, kernel-devel, kernel-modules, and kernel-modules-extra.

Update and reboot

sudo dnf clean all
sudo dnf update -y
sudo reboot

Install virtualization software and add yourself to the user group

sudo dnf install @virtualization
sudo usermod -G libvirt -a $(whoami)
sudo usermod -G kvm -a $(whoami)

Install (proprietary) nVidia drivers and remove/blacklist (open source) nouveau drivers.

sudo su -
dnf install xorg-x11-drv-nvidia akmod-nvidia "kernel-devel-uname-r == $(uname -r)" xorg-x11-drv-nvidia-cuda vulkan vdpauinfo libva-vdpau-driver libva-utils
dnf remove *nouveau*
echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf

Reboot again!
```
sudo reboot
```

Verify that the nVidia drivers are installed correctly.

lsmod | grep nouveau   # This should display nothing
lsmod | grep nvidia    # This should display at least a couple things

Edit /etc/default/grub to enable IOMMU, blacklist nouveau, and load vfio-pci first.

If your video cards are different:

GRUB_CMDLINE_LINUX="rd.driver.pre=vfio-pci rd.driver.blacklist=nouveau modprobe.blacklist=nouveau rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap rhgb quiet intel_iommu=on iommu=pt"

If your video cards are identical, use this instead:

GRUB_CMDLINE_LINUX="rd.driver.pre=vfio-pci rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nvidia-drm.modeset=1 resume=/dev/mapper/arch-swap rd.lvm.lv=arch/root rd.lvm.lv=arch/swap rhgb quiet intel_iommu=on iommu=pt pcie_acs_override=downstream"

Rebuild GRUB's configuration

sudo grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg

Edit /etc/modprobe.d/kvm.conf. These two lines are edited out due to stability concerns. YMMV.
```
#options kvm_intel nested=1
#options kvm_amd nested=1
```
Create or edit /etc/modprobe.d/local.conf, adding the line below:
```
install vfio-pci /sbin/vfio-pci-override.sh
```

Create or edit /etc/dracut.conf.d/local.conf, adding the line below:

add_drivers+= " vfio vfio_iommu_type1 vfio_pci vfio_virqfd "
install_items+=" /sbin/vfio-pci-override.sh /usr/bin/find /usr/bin/dirname "

Create a file /sbin/vfio-pci-override.sh with permissions 755 (file in this directory of the repo).
Rebuild using dracut
```
sudo dracut -f --kver `uname -r`
```
Reboot again!

Verify that your target hardware is using vfio-pci as the driver. Omit the -s 00:02:00 on another machine to get the entire output, as this argument narrows the output down to the device specified.

nataliepc /h/n/k/kernel $ lspci -vv -n -s 00:02:00
02:00.0 0300: 10de:1c82 (rev a1) (prog-if 00 [VGA controller])
  Subsystem: 3842:6251
  Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
  Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
  Interrupt: pin A routed to IRQ 10
  Region 0: Memory at dc000000 (32-bit, non-prefetchable) [disabled] [size=16M]
  Region 1: Memory at a0000000 (64-bit, prefetchable) [disabled] [size=256M]
  Region 3: Memory at b0000000 (64-bit, prefetchable) [disabled] [size=32M]
  Region 5: I/O ports at d000 [disabled] [size=128]
  Expansion ROM at dd000000 [disabled] [size=512K]
  Capabilities: <access denied>
  Kernel driver in use: vfio-pci
  Kernel modules: nouveau, nvidia_drm, nvidia

02:00.1 0403: 10de:0fb9 (rev a1)
  Subsystem: 3842:6251
  Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
  Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
  Interrupt: pin B routed to IRQ 11
  Region 0: Memory at dd080000 (32-bit, non-prefetchable) [disabled] [size=16K]
  Capabilities: <access denied>
  Kernel driver in use: vfio-pci
  Kernel modules: snd_hda_intel

Proceed to set up your virtual machine.

Resources

Alex Williamson's blog on the VFIO tips and tricks - link
Arch Linux wiki post on PCI passthrough - link

Disclaimer

I put this together based on my own machine at home because I knew I'd forget this process if I ever had to do it again. There was a lot of reading of Bugzilla, StackOverflow, and a bunch of blogs/forums/mailing lists all over the internet. Thanks to everyone who did something similar so I could cobble together something from all of them that Works On My Machine. This is by no means the only way to solve the problem. 😃

safizn/fedora-acs-override

Fedora 38 + PCI passthrough

Prerequisites

Setup and configuration of the host machine

Resources

Disclaimer