PCI Passthrough Woes

[Willowwizard]

I'm kinda split on whether I should post this here or in the Troubleshooting forum, but as far as I can tell it seems the Troubleshooting forum is centered mainly on hardware issues so I'll try posting this here instead.

 

So, my computer has intel integrated graphics, and I also have an NVIDIA GTX 1060 3GB slotted in. The integrated graphics is connected via DVI to my monitor, and my dedicated graphics is connected to my monitor via HDMI.

What I'm trying to do is run my Linux installation on the integrated graphics, and then use QEMU's PCI passthrough feature to run Windows on my dedicated graphics card.

 

I currently only have 1 issue.

First, this is more annoying than a critical issue but whenever I boot my computer, I must have the HDMI (dedicated graphics) disconnected, otherwise I get no display ( black screen).

I'm using vfio-pci and blacklisting nouveau in order to get the system to not use my NVIDIA card. The system boots completely normally when only the DVI (integrated graphics) cable is connected.

I've fixed this issue, see my post below.

 

Second, this is more important, whenever my guest VM (Windows 10 via QEMU, using libvirt) is booted, if I have both DVI & HDMI connected and I'm on the DVI input (host integrated graphics), the screen flickers on/off constantly, mostly being a black screen.

If I have both cables connected but the monitor is set to HDMI input (guest dedicated graphics) the flickering isn't present.

If I have only DVI connected, the flickering isn't present and I can see my linux desktop perfectly fine.

 

I'm kind of at a loss here, but I really don't want to have to dual-boot >.< Any suggestions? Please?

[Lord Nicoll]

11 minutes ago, TakataruMC said:

I'm kinda split on whether I should post this here or in the Troubleshooting forum, but as far as I can tell it seems the Troubleshooting forum is centered mainly on hardware issues so I'll try posting this here instead.

 

So, my computer has intel integrated graphics, and I also have an NVIDIA GTX 1060 3GB slotted in. The integrated graphics is connected via DVI to my monitor, and my dedicated graphics is connected to my monitor via HDMI.

What I'm trying to do is run my Linux installation on the integrated graphics, and then use QEMU's PCI passthrough feature to run Windows on my dedicated graphics card.

 

I currently have 2 issues.

First, this is more annoying than a critical issue but whenever I boot my computer, I must have the HDMI (dedicated graphics) disconnected, otherwise I get no display ( black screen).

I'm using vfio-pci and blacklisting nouveau in order to get the system to not use my NVIDIA card. The system boots completely normally when only the DVI (integrated graphics) cable is connected.

 

Second, this is more important, whenever my guest VM (Windows 10 via QEMU, using libvirt) is booted, if I have both DVI & HDMI connected and I'm on the DVI input (host integrated graphics), the screen flickers on/off constantly, mostly being a black screen.

If I have both cables connected but the monitor is set to HDMI input (guest dedicated graphics) the flickering isn't present.

If I have only DVI connected, the flickering isn't present and I can see my linux desktop perfectly fine.

 

I'm kind of at a loss here, but I really don't want to have to dual-boot >.< Any suggestions? Please?

The monitor problems might be due to bad firmware maybe? Having to toggle two inputs, both active should be easy to handle but maybe the monitors firmware is a bit old or corrupted, try upgrading it. The other issue might be Nvidia's lack of (official) PCIe support. I've found GeForce cards only behave relatively well (in this case it's not so well behaved) when using the KVM method, with some virtualisation software's completely not working due to the nvidia drivers hatred of pass through, though it's easy to overcome. 

[Willowwizard]

14 minutes ago, Lord Nicoll said:

The monitor problems might be due to bad firmware maybe? Having to toggle two inputs, both active should be easy to handle but maybe the monitors firmware is a bit old or corrupted, try upgrading it. The other issue might be Nvidia's lack of (official) PCIe support. I've found GeForce cards only behave relatively well (in this case it's not so well behaved) when using the KVM method, with some virtualisation software's completely not working due to the nvidia drivers hatred of pass through, though it's easy to overcome. 

I'm using KVM via QEMU, I've tried looking into upgrading the monitor firmware but there doesn't appear to be any way to do that on my specific monitor (I'm using an IO-DATA DIOS-MF271XDB-B, fair warning however all the information you'll find on this monitor will probably be in Japanese)

I remember being able to use both DVI & HDMI at the same time before (DVI on computer and HDMI from a camera) however so I'm not exactly sure if that's the problem.

[Lord Nicoll]

6 minutes ago, TakataruMC said:

I'm using KVM via QEMU, I've tried looking into upgrading the monitor firmware but there doesn't appear to be any way to do that on my specific monitor (I'm using an IO-DATA DIOS-MF271XDB-B, fair warning however all the information you'll find on this monitor will probably be in Japanese)

I remember being able to use both DVI & HDMI at the same time before (DVI on computer and HDMI from a camera) however so I'm not exactly sure if that's the problem.

Yeah I assumed you where using KVM with QEMU since it was kinda working, but the glitchy display might be a KVM PCIe passthrough work around artifact, thing is since I know it's sketchy, I just buy old firepro or AMD cards as my experience with them on passthrough has been fairly pleasant once Linux isn't involved. You could try passing the iGPU through (idk if that can be done) and see if the screen glitching goes, or try to find a crap AMD card to test that, otherwise I'm not too sure on the error, it is odd it flickers. 

[WereCatf]

This may sound like a stupid question, but I'll ask anyways: have you checked what IOMMU-group the NVIDIA-card is in, and are you passing through all the devices in the IOMMU-group or not?

[Willowwizard]

6 minutes ago, WereCatf said:

This may sound like a stupid question, but I'll ask anyways: have you checked what IOMMU-group the NVIDIA-card is in, and are you passing through all the devices in the IOMMU-group or not?

Nah it's not a stupid question, I didn't really give too much information on the post I guess

Anyway, yes, I've checked the IOMMU groups

IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller [8086:0158] (rev 09)
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] [10de:1c02] (rev a1)
IOMMU Group 1 01:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
IOMMU Group 2 00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller [8086:016a] (rev 09)
IOMMU Group 3 00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)
IOMMU Group 4 00:19.0 Ethernet controller [0200]: Intel Corporation 82579LM Gigabit Network Connection (Lewisville) [8086:1502] (rev 04)
IOMMU Group 5 00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)
IOMMU Group 6 00:1b.0 Audio device [0403]: Intel Corporation 7 Series/C216 Chipset Family High Definition Audio Controller [8086:1e20] (rev 04)
IOMMU Group 7 00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C216 Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)
IOMMU Group 8 00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev a4)
IOMMU Group 9 00:1f.0 ISA bridge [0601]: Intel Corporation C216 Series Chipset LPC Controller [8086:1e53] (rev 04)
IOMMU Group 9 00:1f.2 SATA controller [0106]: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] [8086:1e02] (rev 04)
IOMMU Group 9 00:1f.3 SMBus [0c05]: Intel Corporation 7 Series/C216 Chipset Family SMBus Controller [8086:1e22] (rev 04)

The NVIDIA card is on IOMMU Group 1, I'm using vfio-pci drivers on 10de:1c02 and 10de:10f1 (VGA & HDMI audio, respectively), and I'm passing both through to the VM. The only thing that isn't using the vfio-pci driver and isn't being passed to the VM is the PCI bridge device, which, as I understand, is required to be available to the host, I think?

 

EDIT: Also, just in case; yes, I've confirmed that both are using the vfio-pci drivers

[ludburgh@marvin ~]$ lspci -nnk -d 10de:1c02
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] [10de:1c02] (rev a1)
        Subsystem: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] [10de:11c2]
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau
[ludburgh@marvin ~]$ lspci -nnk -d 10de:10f1
01:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
        Subsystem: NVIDIA Corporation GP106 High Definition Audio Controller [10de:11c2]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel
[ludburgh@marvin ~]$ 

 

[WereCatf]

2 minutes ago, TakataruMC said:

The only thing that isn't using the vfio-pci driver and isn't being passed to the VM is the PCI bridge device, which, as I understand, is required to be available to the host, I think?

AFAIK, you're supposed to pass through all the devices in the group, including the bridge. Shouldn't hurt to give it a try, I would think.

[Willowwizard]

I was able to solve the first problem, I added the video=efifb:off kernel parameter to set the text console output to my iGPU, and made an xorg configuration file (in /etc/X11/xorg.conf.d/20-intel.conf ) to configure Xorg to use the iGPU

Simply adding

Section "Device"
  Identifier  "Intel Graphics"
  Driver      "intel"
EndSection

did the trick.
Now I can boot with the dGPU's HDMI connected and have Linux display to the iGPU output

[Willowwizard]

2 minutes ago, WereCatf said:

AFAIK, you're supposed to pass through all the devices in the group, including the bridge. Shouldn't hurt to give it a try, I would think.

Alright, I'll try that and report back

[Willowwizard]

Alright, the VM won't even start if I try passing through the PCI bridge

Error starting domain: internal error: Non-endpoint PCI devices cannot be assigned to guests

I've followed the tutorial on the ArchWiki, (https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Using_vfio-pci), and in the vfio-pci section it says that I shouldn't pass the PCI root port (PCI bridge) even if it's a part of the IOMMU group

[Fleetscut]

9 hours ago, TakataruMC said:

Alright, the VM won't even start if I try passing through the PCI bridge

Error starting domain: internal error: Non-endpoint PCI devices cannot be assigned to guests

I've followed the tutorial on the ArchWiki, (https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Using_vfio-pci), and in the vfio-pci section it says that I shouldn't pass the PCI root port (PCI bridge) even if it's a part of the IOMMU group

the problem is you need to pass through everything that is in the same iommu group. you cant pass through only one component. your pci bridge is in the same group as your gpus graphics and audio components, so you wont be able to pass through just the gpu. What you need to do is the use an ACS patched kernel. usually you would need to do this yourself, however if you are on arch this has been put onto the AUR. use an AUR helper to get the linux-vfio and linux-vfio-headers packages. then rerun grub config and boot with the new kernel. once using this kernel all PCI devices are put into thier own iommu group.

 

once you do get your VM up an running and your passthrough working you are going to run into another problem. NVIDIA does not allow their drivers to run if it detects a hypervisor is being used to host the OS. if you go into device manager in your windows vm and see that the graphics card is showing an Error 43 code then it means that you have this problem. you will need to edit your KVM config and add hidden state = on.

 

 

[Willowwizard]

5 hours ago, Fleetscut said:

the problem is you need to pass through everything that is in the same iommu group. you cant pass through only one component. your pci bridge is in the same group as your gpus graphics and audio components, so you wont be able to pass through just the gpu. What you need to do is the use an ACS patched kernel. usually you would need to do this yourself, however if you are on arch this has been put onto the AUR. use an AUR helper to get the linux-vfio and linux-vfio-headers packages. then rerun grub config and boot with the new kernel. once using this kernel all PCI devices are put into thier own iommu group.

 

once you do get your VM up an running and your passthrough working you are going to run into another problem. NVIDIA does not allow their drivers to run if it detects a hypervisor is being used to host the OS. if you go into device manager in your windows vm and see that the graphics card is showing an Error 43 code then it means that you have this problem. you will need to edit your KVM config and add hidden state = on.

 

 

I've already tried that kernel, and already set up my VM properly (with NVIDIA drivers & configuring VM to change the hardware identifier), the issue is not that the VM isn't starting or recognizing the card, it works fine, I just can't view my host display properly while the VM is on, the Windows display works fine, but the Linux display starts flickering and generally acting crazy as long as the VM is started

[Willowwizard]

Okay, I feel like an idiot now, I've fixed it by swapping out the HDMI cable with a new one.

I don't know how a bad HDMI cable would affect the DVI display, maybe a bad ground connection or something perhaps?

Anyway, at least it's fixed now

 

Thanks to everyone who took the time to try helping me, I really appreciate it!

[Fleetscut]

On 7/22/2018 at 10:24 PM, TakataruMC said:

Okay, I feel like an idiot now, I've fixed it by swapping out the HDMI cable with a new one.

I don't know how a bad HDMI cable would affect the DVI display, maybe a bad ground connection or something perhaps?

Anyway, at least it's fixed now

 

Thanks to everyone who took the time to try helping me, I really appreciate it!

Ah, i think i misunderstood you. Was it stuck in a boot loop? I has this once too and went nuts trying to diagnose. Mine also turned out to be a cable issues. Odd thing is it worked fine on the host, only had problems on the guest.

[Willowwizard]

14 hours ago, Fleetscut said:

Ah, i think i misunderstood you. Was it stuck in a boot loop? I has this once too and went nuts trying to diagnose. Mine also turned out to be a cable issues. Odd thing is it worked fine on the host, only had problems on the guest.

No it wasn't stuck in a boot loop, the issue was that when the guest boots (i.e. HDMI cable has something going through it) my host display (i.e. signal going through DVI cable) would start flickering, glitching, losing signal, generally become unusable, turns out it was just a bad HDMI cable