Passthrough issues with nvidia GPU + XP VM

[marioawe]

Hi there,

Running into an issue with a build I am setting up, specifically passing through a GPU through to a XP VM being used offline, for retro gaming. The GPU does get recognized correctly inside the VM, but fails to start with an Error code 10. I've gone through forums and tried several things, all to no avail.

Anyone know what I'm doing wrong here? Please let me know if I can provide any other info!

System:

CPUs	2x E5-2690 v2
Motherboard	Supermicro X9DRi-LN4F+
GPU (for modern OS)	GTX 1050
GPU (for retro OS)	GT 960
RAM	Samsung, 96GB total

/etc/default/grub

GRUB_DEFAULT=0 GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo 
Debian` GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt intel_iommu=on 
initcall_blacklist=sysfb_init" GRUB_CMDLINE_LINUX=""

/etc/modules

vfio  vfio_iommu_type1  vfio_pci

/etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:1d01,10de:0fb8,10de:104a,10de:0e08 disable_vga=1 
softdep nouveau pre: vfio-pci 
softdep snd_hda_intel pre: vfio-pci

/etc/modprobe.d/blacklist.conf

blacklist nouveau 
blacklist nvidia* 
blacklist nvidiafb

/etc/pve/qemu-server/XPNUMHERE.conf

## SP Version:
#Service Pack 3
acpi: 1
agent: 0,type=isa
args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
boot: order=ide0;ide1;ide2
cores: 2
cpu: host,hidden=1,flags=+pcid
hostpci0: 0000:04:00.0
hostpci1: 0000:04:00.1
ide0: local-lvm:vm-202-disk-0,cache=writeback,size=60G
ide1: local:iso/en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso,media=cdrom,size=603276K
ide2: local:iso/virtio-win-0.1.240.iso,media=cdrom,size=612812K
machine: pc-i440fx-8.1
memory: 4096
meta: creation-qemu=8.1.5,ctime=1708923883
name: WinXP-Games
net0: virtio=BC:24:11:5F:32:C2,bridge=vmbr0,firewall=1
numa: 0
ostype: wxp
scsihw: virtio-scsi-single
smbios1: uuid=fd3f66fa-420f-4243-b51c-9c99afd7f293
sockets: 1
tags: Games;Windows
vmgenid: 1a742af8-73a0-4e8a-9aae-da6c159294b7

 

[Windows7ge]

Did you check with lspci -vnn to verify both the video and audio devices are using the vfio-pci kernel driver?

 

5e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107GL [Quadro P1000] [10de:1cb1] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Hewlett-Packard Company GP107GL [Quadro P1000] [103c:11bc]
        Flags: bus master, fast devsel, latency 0, IRQ 993, NUMA node 0, IOMMU group 92
        Memory at c4000000 (32-bit, non-prefetchable) [size=16M]
        Memory at a0000000 (64-bit, prefetchable) [size=256M]
        Memory at b0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at 9000 [size=128]
        Expansion ROM at c5000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] Secondary PCI Express
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau

5e:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
        Subsystem: Hewlett-Packard Company GP107GL High Definition Audio Controller [103c:11bc]
        Flags: bus master, fast devsel, latency 0, IRQ 992, NUMA node 0, IOMMU group 92
        Memory at c5080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

 

[marioawe]

1 minute ago, Windows7ge said:

Did you check with lspci -vnn to verify both the video and audio devices are using the vfio-pci kernel driver?

 

5e:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107GL [Quadro P1000] [10de:1cb1] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Hewlett-Packard Company GP107GL [Quadro P1000] [103c:11bc]
        Flags: bus master, fast devsel, latency 0, IRQ 993, NUMA node 0, IOMMU group 92
        Memory at c4000000 (32-bit, non-prefetchable) [size=16M]
        Memory at a0000000 (64-bit, prefetchable) [size=256M]
        Memory at b0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at 9000 [size=128]
        Expansion ROM at c5000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [420] Advanced Error Reporting
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] Secondary PCI Express
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau

5e:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
        Subsystem: Hewlett-Packard Company GP107GL High Definition Audio Controller [103c:11bc]
        Flags: bus master, fast devsel, latency 0, IRQ 992, NUMA node 0, IOMMU group 92
        Memory at c5080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

 

That is correct, here is the relevant output:

 

 

I also explicitly have it so that it loads vfio-pci before nvidia's drivers in my /etc/modprobe.d/vfio.conf:

options vfio-pci ids=10de:1d01,10de:0fb8,10de:104a,10de:0e08 disable_vga=1 
softdep nouveau pre: vfio-pci 
softdep snd_hda_intel pre: vfio-pci

 

[Windows7ge]

21 hours ago, marioawe said:

That is correct, here is the relevant output:

 

 

I also explicitly have it so that it loads vfio-pci before nvidia's drivers in my /etc/modprobe.d/vfio.conf:

options vfio-pci ids=10de:1d01,10de:0fb8,10de:104a,10de:0e08 disable_vga=1 
softdep nouveau pre: vfio-pci 
softdep snd_hda_intel pre: vfio-pci

Did you enable VT-d?

How about did you set the VM to use the Q35 chipset instead of the 440FX?

[marioawe]

45 minutes ago, Windows7ge said:

Did you enable VT-d?

How about did you set the VM to use the Q35 chipset instead of the 440FX?

VT-d is enabled. I have not had success installing WinXP on a Q35 chipset as it BSODs right after it loads files and attempts to start Windows. With 440FX I do not encounter these issues.

[m9x3mos]

1 hour ago, marioawe said:

VT-d is enabled. I have not had success installing WinXP on a Q35 chipset as it BSODs right after it loads files and attempts to start Windows. With 440FX I do not encounter these issues.

Does it work with windows 10 vms?

I think that the card is probably too new for windows xp to know what to do with it even if passthrough is working. 

[marioawe]

1 hour ago, m9x3mos said:

Does it work with windows 10 vms?

I think that the card is probably too new for windows xp to know what to do with it even if passthrough is working. 

It does work with Win10. Both the initial GT 610 and the GTX 960 I just acquired should not be too new for XP, drivers exist from nvidia specifically for XP 32 bit for both, latest release in 2016.

 

Drivers:

https://www.nvidia.com/download/driverResults.aspx/105040/en-us/ 

 

[Windows7ge]

5 hours ago, marioawe said:

VT-d is enabled. I have not had success installing WinXP on a Q35 chipset as it BSODs right after it loads files and attempts to start Windows. With 440FX I do not encounter these issues.

 

I'm not familiar with i440FX supporting hardware pass-though. Tutorials I've followed myself have you switch to Q35 because of a virtual PCI_e feature that's included.

I know Microsoft released a 64-bit version of XP but I don't know if that would help any.

[marioawe]

10 hours ago, Windows7ge said:

I'm not familiar with i440FX supporting hardware pass-though. Tutorials I've followed myself have you switch to Q35 because of a virtual PCI_e feature that's included.

I know Microsoft released a 64-bit version of XP but I don't know if that would help any.

Admittedly, neither do I. I may try using the x64, but I have limited experience with it regarding XP - I was trying to stick to 32bit as the majority of the games are from that period, and were originally played on an old chonky Dell Dimension 3000.

 

That all being said - I've been digging into using Q35 and it seems that BSOD comes up when it doesn't recognize the storage controller. I'm relatively familiar with slipstreaming drivers into ISOs with nLite/NTLite and added the drivers from the viostor folder from the VirtIO Windows driver ISO. This causes it to not prompt me that it doesn't see the storage controller, but still ultimately fails out with the same BSOD. Attempting any of the other controllers, the installer obviously doesn't seem to see them.

[Windows7ge]

6 hours ago, marioawe said:

Admittedly, neither do I. I may try using the x64, but I have limited experience with it regarding XP - I was trying to stick to 32bit as the majority of the games are from that period, and were originally played on an old chonky Dell Dimension 3000.

 

That all being said - I've been digging into using Q35 and it seems that BSOD comes up when it doesn't recognize the storage controller. I'm relatively familiar with slipstreaming drivers into ISOs with nLite/NTLite and added the drivers from the viostor folder from the VirtIO Windows driver ISO. This causes it to not prompt me that it doesn't see the storage controller, but still ultimately fails out with the same BSOD. Attempting any of the other controllers, the installer obviously doesn't seem to see them.

I've never tried pass-through on an XP VM.  Are the error details any more descriptive than error code 10? Proxmox should tell you more in the log.

 

Did you explore any of the parameters you can set? Maybe you can pass-through on i440FX but you have to make the VM think it's a PCI device not PCIe.