Nvidia GPU passthrough driver issues

[Mathew_96]

Hey folks,

I'm having a bit of an issue with my server and where could be a better place to ask for help than here? :D

So after having seen all of the X-Gamers 1-CPU videos I thought I might try aswell...
A bit of backstory:

I happen to have an HP-DL580 gen 7 idling around with an installation of xcp-ng (a free version of XEN) I hosted multiple VMs at Haxogreen (local hacker event) and it did it's job pretty well.
So I wanted to use this as base for my test to play with two people on one machine. 

I used XCPs GPU-Passthrough to hand two cards to two VMs (Gtx 550 Ti and Quadro K420) I know these aren't baller GPUs but I wanted to see if it works first.

Well: It doesn't

Windows recognizes the cards, driver installs but on restart "Error 43"

Google tells me this is because NVIDIA drivers refuse to load if they detect they are running in a virtualized environment. I found some solutions for unRaid like disabling Hyper-V or even patching the driver. 
Other solutions include spoofing the hv_vendor_id for kvm.

 

Does anybody knnow how to do something like this in XCP/XEN? Ideally without installing another host system?

Thanks in advance,

Mathew_96

[Salv8 (sam)]

so long as your virtualization software allows the option of telling the virtual machine that it's not a virtual machine, then the drivers should install without any mods needed/required

unfortunately from what i could find, this isn't yet supported with your virtualization software and does require a patched nvidia driver

(https://xcp-ng.org/forum/topic/295/xcp-ng-gpu-passthrough-no-gpu-found/13)

on the unraid fourms the user "Siwat2545" has done a guide on how to get this working, on unraid...

( https://forums.unraid.net/topic/69670-guide-fix-nvidia-code-43-issue-on-nvidia-gpu/ )

you are best off asking on the xcp-ng forums on how to do this as they will have more experience in getting it working

[Mathew_96]

Hey, thanks for the answer .

I also stumbled upon this thread from Siwat2545 and I think I'll try to patch the driver.
XCP forums might be a good idea somehow did not come up with that -.-

If it won't work I think I need to switch to something like unRaid... that would not be a total desaster but if I can avoid it....

[Mathew_96]

So for those who are still searching there are only two options for XEN/XCP with Nvidia GPUs from waht I found on XCP-Forums: you can either patch the driver (I will try that) or patch XEN/XCP to "mask" itself like KVM can do

( https://lists.xenproject.org/archives/html/xen-devel/2016-07/msg01713.html )

Note: this patch is not for the current version of XCP you might need to modify it.

I'll report back if I had success with patching the drivers...

[Mathew_96]

Okey with the patched driver I'm getting a bluescreen at boot, does not seem to work.

But I will try with an AMD card as soon as I have one to test that.

[Mathew_96]

Okey with an RX480 I get bluescreen when installing the Driver... damn. And it is not the PSU I tried with 400W psu externally connected to only the RX480...

[overlinus]

Hi, Mathew, did you managed to get it working?

[Mathew_96]

Sadly not, I had to give the AMD GPU back, (was not mine) so I could not test any further. I will continue to try with the NVIDIA cards, but I think I will have to patch XCP... or use something else like proxmox? or unRaid? (unRaid HAS TO WORK XD)

[Mikensan]

If you're willing to switch to ESXi (also free and widely loved) you could try this: 

I believe there's also a way within windows registry to modify how it reports hardware, I actually came across it on youtube watching people scam scammers... apparently scammers look for VMs by looking at device manager, so somebody figured out how to mask it - might work depending on how nvidia queries.

 

 

[Stancey]

You may be seeing compatibility issues with the Keppler architecture chipset.  When I tried running with the Tesla K80 there was a requirement that on a given bus that only Keppler Architecture GPUs be present when using the driver. It seems there is a reluctance on the part of the Keppler support drivers to share the bus control with other graphics cards.  There is a warning in the Tesla K80 documentation to only use Keppler Architecture graphics cards with the Tesla.  Quadro is based on the Keppler Architecture . On that not be aware that Nvidia recently came out with some updates to their drivers that support the Tesla/Quadro pairing. I think it was in August or September.  The one I have is working much better with the most recent drivers. 

[Stancey]

On ‎11‎/‎22‎/‎2018 at 8:59 AM, Mathew_96 said:

Sadly not, I had to give the AMD GPU back, (was not mine) so I could not test any further. I will continue to try with the NVIDIA cards, but I think I will have to patch XCP... or use something else like proxmox? or unRaid? (unRaid HAS TO WORK XD)

I did try to pair the Keppler with an AMD and it kept crashing the kernel. 

[Mathew_96]

Hey, the solution with ESXi seems fairly simple... I think I'll check what system I can transfer my current VMs to most easily.

 

16 hours ago, Stancey said:

I did try to pair the Keppler with an AMD and it kept crashing the kernel. 

Interresting, sadly I don't have another AMD card at the moment but I'll remove the Quadro and test again with the patched driver...