[GUIDE] Linux PCI GPU VFIO Passthrough

October 2, 2018

So, you've probably seen the gaming on Linux Linus/Wendell video, and you probably figured out how hard it is to make it actually work through PCI passthrough, especially when searching on google how to do so, most of the guides are outdated and they don't explain too much. Exception obv.
I'm here to explain the simplest method available. I'm not covering everything, just the basis. You still need to search some specific workaround by yourself, of course I will tell you where to search and the most updated materials.

Prerequisites

Motherboard I/O Virtualization support
CPU with Intel VT-d or AMD-Vi support, and option enabled in the BIOS
Two video cards (Can be an iGPU as well)
Desktop PC (Some laptops are working but the list is sooo small, refer to this)
Knowing what you are doing, this is not 100% guaranteed to work
GNU/Linux distro with a decent kernel version
Decent PC knowledge, basic Linux command line usage (You though you could do this without touching the terminal? Too bad.)
UEFI bios and UEFI GPU

If you don't meet those requirements you can stop reading, go back using windows if you care playing games (bad joke, not even funny)

Recommended setup

Intel CPU (AMD has some issues getting the right core count to work in QEMU)
NVIDIA GPU (Does not suffer from soft reset issues)
Dedicated PCI slot lanes (Will explain that later)

Useful materials

Google
https://level1techs.com/article/ryzen-gpu-passthrough-setup-guide-fedora-26-windows-gaming-linux
Arch wiki, https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVM
https://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/
https://passthroughpo.st/explaining-csm-efifboff-setting-boot-gpu-manually/
https://discord.gg/hqrRDXG Discord invite to the VFIO server (Not mine) it has also a lot of materials, and you can ask people for help in there.

Getting started

PCI Passthroughing is available on Linux thanks to vfio and KVM, and gives the possibility to attach PCI devices like GPUs.
We basically want to play games on a virtualized Windows virtual machine by dedicating one PCI GPU slot to the virtual machine, this is possibile thanks to VFIO. The virtualized environment we are going to use is the KVM Hypervisor, and the actual Linux programs to use are called QEMU and libvirt (libvirt supports more hypervisors, but we are still going to use KVM), we are going to use libvirt + virt-manager which is the Red Hat linux solution to make it easier to manage virtual machines. Why? Because it has a GUI called virt-manager libvirt has a different way to manage VM settings through XML which is different from the QEMU bash style and has a lot of features.

First of all, install libvirt and qemu, in ubuntu you can do that by

sudo apt-get install virt-manager ovmf

(Will also install libvirt)
Don't want to use the terminal? Good. You can still use the GUI but that is way faster and less frustrating, so stop crying like a little baby. All OS have a command line too. You can use synaptic or the ubuntu store to install programs, did you know?

Checking if IOMMU is correctly loaded and enabled

dmesg | grep IOMMU

This should give us something like

[    0.000000] Intel-IOMMU: enabled
[    0.028879] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.028883] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da
[    0.536229] IOMMU 0 0xfed90000: using Queued invalidation
[    0.536230] IOMMU 1 0xfed91000: using Queued invalidation
[    0.536231] IOMMU: Setting RMRR:
[    0.536241] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf000000 - 0xcf1fffff]
[    0.537490] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdea8000 - 0xbdeb6fff]
[    0.537512] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdea8000 - 0xbdeb6fff]
[    0.537530] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdea8000 - 0xbdeb6fff]
[    0.537543] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.537549] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

If you have problems, check if IOMMU is enabled in the bios, either you have unsupported CPU or MOBO. You can force Linux to load iommu by putting those kernel parameters in /etc/default/grub `intel_iommu=on` for intel CPUs or `amd_iommu=on` for AMD

Checking IOMMU groups

#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do 
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;

This is a simple bash script that should tell us which IOMMU groups are available, search for your GPU, if your GPU IOMMU group shares something with other devices, you must pass them as well. More on that later.

IOMMU Group 1 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
IOMMU Group 1 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)
IOMMU Group 1 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)

This is an output example, you would need to pass everything on that group. Having an audio device shared with the GPU makes sense because it is the HDMI audio of your GPU, you should start to worry if you are seeing other devices.

There are several ways to try separating the GPU from an IOMMU group, like swapping the PCI-e slot with another one, then run again that command.
If you want to mess up your system, you can use ACS but I'm not covering that, refer to the arch wiki please. It is however complicated, you'll need to patch your kernel. If you are on arch you can use the linux-vfio one from the AUR, but note that this just makes the kernel trick IOMMU groups are all separated each other for each device when they are not, strange things may happen

Passing the actual PCI devices

Now you are almost good to go, what you are gonna need is to enable vfio from in the ramdisk to ensure it will load on the kernel startup.
So you need on Ubuntu to edit the following file /etc/initramfs-tools/modules as root and add those drivers

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

On arch the procedure is similiar, it is just a different file. After that, update your initramfs (just an ubuntu thing) by

sudo update-initramfs -u

Obviously they will stop working from your host PC when you unbind them.

Now the passing procedure consists in editing a file, we'll call it vfio.conf, in /etc/modprobe.d/vfio.conf

options vfio-pci ids=xxxx:yyyy,xxxx:yyyz

where the ids are your hardware ids you need to pass through, you can find them by typing `lspci -nn` and then searching for your required devices needed from into the IOMMU group, most of the time you just need the ID of the graphics card and the graphics card audio id.

Its format is xxxx:yyyy as you can see from that picture. Different ids needs to be separated with a comma
If you still can't get it working, go to the troubleshooting section.

If everything went smooth, reboot and the vfio-pci driver should have binded to your card, you can check this by typing `sudo lspci -v`
The result should be something like

1c:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] (rev a1) (prog-if 00 [VGA controller])
        ...
        Kernel driver in use: vfio_pci
        ...

In that case, the GPU binding was successful. Check this for all your devices you need to pass.

Virtual machine configuration

Now this should be the easiest part, once you passed your devices to the vfio driver, we just need to build a decent virtual machine.
As I said earlier, this could be done by just using plain qemu, but I'm doing the libvirt thing using virt-manager, so once you installed the virt-manager program, open it! (You don't even need terminal commands...To start it)
But too bad;

before opening it, we need to first edit /etc/libvirt/qemu.conf as sudo

Uncomment this whole part of text by just removing the "#", we need that to enable the UEFI bios in the virtual machine.
Reboot your PC after saving that file. Proceed to the virtual machine configuration wizard.
Configure as the following.

Configure the vm as you want, the most important thing is to choose the UEFI firmware.

Start your Windows installation.

Now it is time to get serious.

Time to add the video card and all the devices you passed to vfio

Don't mind my video card, I'm just showing how to do that on another PC. You need to pass your NVIDIA GPU and all the other devices required for your IOMMU group.

Note that to check if the video card is working, you should check its video output.
What you have on the virtual machine window is an emulated software video card.

Start your vm
If you have an NVIDIA gpu and from the Windows devices manager you are getting "error 43" its normal. You need to hide the windows NVIDIA driver you are using a VM.
You can to that by editing /etc/libvirt/qemu/yourvmname.xml

This is like editing an HTML file. You need to edit the file as the following

...
<features>
	<hyperv>
		...
		<vendor_id state='on' value='1234567890ab'/>
		...
	</hyperv>
	...
	<kvm>
	<hidden state='on'/>
	</kvm>
</features>
...

Just make sure you are on the right section. vendor_id state must me inside the <hyperv> and kvm inside <features>
Save the file, you may need to reboot your PC for the changes to be applied. If something is wrong with the syntax, starting virt-manager will not show the vm in the list.

That's all for the basic configuration

Final notes

You may want to disable the emulated Cirrus or QXL software video card to just use the dedicated video card, I repeat, just check the video card output. You can choose other ports on your monitor. And there you will have your virtualized Gaming OS! Also, you may not want to use the virtualized mouse pointer and the keyboard, you can choose to pass-through the hardware one by just putting the mouse like you did with the GPU, just use USB instead of PCI you may not need to use vfio. Just use virt-manager. Just make sure to have another mouse unplugged for the VM, or you ha

Reboot, and now the graphics card should be working and everything should be fine.

I'm not covering LookingGlass since that this guide is meant to be intended just for the basic configuration. I'm not saying I won't add it in the future.

Troubleshooting
As for other configurations here is a list of issues and solutions. First of all, you should check the arch wiki. It has really a good list. I can't really cover every thing. But I will try my best.

Still error 43: First of all, check the video card output, if you see the OVMF bios logo, there is something misconfigured with the QEMU config as you need to hide KVM as I said before.
If there is a blank screen, you may probably need to extract your GPU vbios. This is required on some configurations. I personally needed it for example.
You can use nvflash for that purpose. The syntax is `./nvflash_linux --save vbios.rom` Once you extracted the bios, if the IFR Size is more than 0, it needs to be removed. If the IFR size if 1536 byte, for example, remove it by using dd, the syntax is `dd if=vbios.rom of=vbios.fixed.rom bs=1536 skip=1`
Now you can put your GPU bios into libvirt by editing the .xml, first of all under your GPU <hostdev> put the following

<hostdev>
     ...
     <rom file='/path/to/your/gpu/bios.bin'/>
     ...
   </hostdev>

Be sure the .bin file is readable. That's it, save and reboot.

Wrong CPU core count on AMD cpus: That is a known issue, there is no clear solution. You may choose older CPU configuration like Skylake or Core2duo and have at least 4 core count enabled on windows. You can also try to update QEMU to version 3 as there seem to be a fix, that basically consist to make changes to the the xml file to something like

<cpu mode='host-passthrough' check='none'>
  <topology sockets='1' cores='8' threads='2'/>
  <cache mode='passthrough'/>
  <feature policy='require' name='topoext'/>
</cpu>

For a ryzen 7 that should be enough.
On older versions of QEMU you could try the solution listed there http://mathiashueber.com/amd-ryzen-based-passthrough-setup-between-xubuntu-16-04-and-windows-10/ There are also some Ryzen tricks.

Still can't bind the GPU to vfio: There may be the efi framebuffer that uses the primary GPU, maybe it is the first PCI slot? Try with the kernel parameter `video=vesafb:off,efifb:off` by editing /etc/default/grub

That's just it for now I will obviously fix any error present, please suggest changes if you find something wrong.