Skeen/libvirt-gpu-passthrough

blind mode

eoli3n opened this issue · 8 comments

eoli3n commented

Hi,
I found on archlinux wiki that you faced blind mode.
What did you do to solve it ?

I'm trying to pci passtrhough guest archlinux host archlinux, and after selecting my grub entry i get this.

Loading Linux linux ...
Loading initial ramdisk ...
error: no suitable video mode found.
Booting in blind mode
eoli3n commented

did you have a font problem ?

virt_003

Skeen commented

Hi @eoli3n,

I made two changes to get around the problem:

I used vendor_id spoofing, as described in the "Error 43: Driver failed to load" on Nvidia GPUs passed to Windows VMs of https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Troubleshooting, I believe this might only be required for windows.

And the second change I did, was to load a modified VBios into the GPU, to ensure GOP/UEFI support. Can you verify that your card does in fact have GOP/UEFI support?

eoli3n commented

Thx for your quick answer !

I used vendor_id spoofing, as described in the "Error 43: Driver failed to load" on Nvidia GPUs passed to Windows VMs of https://wiki.archlinux.org/index.php

My guest is Archlinux and not Windows, but yes i tried that workaround in vm xml file without success.

Can you verify that your hard does in fact have GOP/UEFI support?
It seems that i can't, i want to extract it then test with https://github.com/awilliam/rom-parser

I want also try to follow https://github.com/jscinoz/optimus-vfio-docs

Nvidia VBIOS must be provided to the guest via libvirt's / qemu's romfile option on the device

Both need my gpu rom, lets try to extract it

❯ dmesg | grep -i vfio
[    0.000000] Command line: initrd=\intel-ucode.img initrd=\initramfs-linux.img rd.luks.uuid=e3a6a7af-5093-4d74-8d42-2f32b7e5fea6 rd.lvm.lv=universe/lvroot rd.lvm.lv=universe/lvswap rd.luks.options=discard rd.luks.options=timeout=0 rootflags=x-systemd.device-timeout=0 root=UUID=1151417d-47d4-40a2-982f-21b0afe4a4aa rw threadirqs i915.enable_ips=0 video.use_native_backlight=1 elevator=noop resume=/dev/universe/lvswap quiet splash vt.global_cursor_default=0 intel_iommu=on iommu=pt vfio-pci.ids=10de:1341 vfio-pci.disable_idle_d3=1 disable_vga=1 video=efifb:off
[    0.041293] Kernel command line: initrd=\intel-ucode.img initrd=\initramfs-linux.img rd.luks.uuid=e3a6a7af-5093-4d74-8d42-2f32b7e5fea6 rd.lvm.lv=universe/lvroot rd.lvm.lv=universe/lvswap rd.luks.options=discard rd.luks.options=timeout=0 rootflags=x-systemd.device-timeout=0 root=UUID=1151417d-47d4-40a2-982f-21b0afe4a4aa rw threadirqs i915.enable_ips=0 video.use_native_backlight=1 elevator=noop resume=/dev/universe/lvswap quiet splash vt.global_cursor_default=0 intel_iommu=on iommu=pt vfio-pci.ids=10de:1341 vfio-pci.disable_idle_d3=1 disable_vga=1 video=efifb:off
[    1.636764] VFIO - User Level meta-driver version: 0.3
[    1.647177] vfio_pci: add [10de:1341[ffffffff:ffffffff]] class 0x000000/00000000
[   11.333369] vfio-pci 0000:03:00.0: enabling device (0006 -> 0007)
[   11.333437] bbswitch: device 0000:03:00.0 is in use by driver 'vfio-pci', refusing OFF

when i start my VM i get

[  106.944786] vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258
[  106.944808] vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[  106.951080] vfio-pci 0000:03:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x0000
[  109.988538] vfio-pci 0000:03:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x0000

Searching Invalid PCI ROM header signature: expecting 0xaa55, got 0x0000 on web leads to https://www.redhat.com/archives/vfio-users/2019-March/msg00004.html

root@osz ~# echo 1 > /sys/bus/pci/devices/0000:03:00.0/enable
write: Périphérique ou ressource occupé
root@osz ~ [1]# echo 1 > /sys/bus/pci/devices/0000:03:00.0/rom
root@osz ~# cat /sys/bus/pci/devices/0000:03:00.0/rom > gpu.rom
cat: '/sys/bus/pci/devices/0000:03:00.0/rom': Erreur d'entrée/sortie

# just after reboot

root@osz ~# lspci -s 03:00.0 -vvv
03:00.0 3D controller: NVIDIA Corporation GM108M [GeForce 840M] (rev a2)
	Subsystem: Dell GM108M [GeForce 840M]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

# no "-Mem"

root@osz ~# echo 1 > /sys/bus/pci/devices/0000:03:00.0/enable
write: Périphérique ou ressource occupé
root@osz ~ [1]# echo 1 > /sys/bus/pci/devices/0000:03:00.0/rom
root@osz ~# cat /sys/bus/pci/devices/0000:03:00.0/rom > gpu.rom
cat: '/sys/bus/pci/devices/0000:03:00.0/rom': Erreur d'entrée/sortie

# then fire VM

root@osz ~# lspci -s 03:00.0 -vvv
03:00.0 3D controller: NVIDIA Corporation GM108M [GeForce 840M] (rev a2)
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

# no changes

root@osz ~# setpci -s 03:00.0 COMMAND=2:2
root@osz ~# echo 1 > /sys/bus/pci/devices/0000:03:00.0/enable
write: Périphérique ou ressource occupé
root@osz ~ [1]# echo 1 > /sys/bus/pci/devices/0000:03:00.0/rom
root@osz ~# cat /sys/bus/pci/devices/0000:03:00.0/rom > gpu.rom

My GPU is an Optimus one : https://wiki.archlinux.org/index.php/Bumblebee#Power_management
That's why i get in dmesg | grep vfio, that line

[   11.333437] bbswitch: device 0000:03:00.0 is in use by driver 'vfio-pci', refusing OFF

bbswitch manage GPU power. Maybe that is why i can't get -Mem at startup.

Skeen commented

I haven't tried dumping the VBIOS from Linux, as written in my wiki entry, I utilized GPU-Z + GOPupd, both of which I ran in Windows.

Even though it's blasphemic to suggest, you could try a similar approach. - I believe I did it all through the SPICE video output, as the card did not boot in Windows at this point.

eoli3n commented

i'm trying to extract it from bios upgrade with https://github.com/coderobe/VBiosFinder

eoli3n commented 
~/vbios/VBiosFinder master*
❯ ./vbiosfinder extract ~/downloads/E7450A23.exe
output will be stored in '/home/user/vbios/VBiosFinder/tmp-vbiosfinder'
checking for ruby... yes

checking for innoextract... yes
checking for upx... yes
checking for 7z... yes
trying to extract ./E7450A23.exe
found polyglot archive
trying to extract ./E7450A23.exe-polyglot
found polyglot archive
trying to extract ./E7450A23.exe-polyglot-polyglot
found polyglot archive
trying to extract ./E7450A23.exe-polyglot-polyglot-polyglot
found zlib archive
trying to extract ./E7450A23.exe-polyglot-polyglot-polyglot-zlib
extracting uefi data
trying to extract ./E7450A23.exe
found UEFIExtract archive
trying to extract ./E7450A23.exe-polyglot
found UEFIExtract archive
trying to extract ./E7450A23.exe-polyglot-polyglot
found UEFIExtract archive
trying to extract ./E7450A23.exe-polyglot-polyglot-polyglot
found UEFIExtract archive
trying to extract ./E7450A23.exe-polyglot-polyglot-polyglot-zlib
found UEFIExtract archive
trying to extract ./mkmf.log
found UEFIExtract archive
filtering for modules...
got 264 modules
finding vbios
no candidates found :(
input contains uefi data but no vbios could be found
the vbios might not be baked into the input!
Cleaning up garbage

next...