ESXi GPU passthrough

[SteveiJobs]

Hey everyone,

 

I'm working on an application where I'm passing a GPU (Nvidia RTX 6000 passive) into a VM in ESXi 7u2. After many hours of struggling just getting the VM to boot (hint, there's not much good literature on what the memory mapping value needs to be), I got the VM to boot and installed the drivers and everything is working great!

 

Except...

 

The VM refuses to use the Nvidia GPU and just defaults to the VMWare SVGA 3D driver and won't use the Nvidia card for Blender, games, or anything else that I'm trying to GPU accelerate. I have tried disabling and uninstalling the SVGA 3D driver but no dice (VMWare Tools seems to put it right back). I feel like it might be related to the fact that the card is a server variant with no display outputs and is meant to be used for compute (which it will also be used for on a later project) but all the Nvidia documentation indicates that it can be used for display acceleration for things like Horizon View. 

 

Other relevant info: Server is a PowerEdge R7525 with 2x Epyc 7302 CPUs and Guest OS is Server 2019. I tried both the "Datacenter" and "Quadro" drivers from Nvidia.

 

Anybody have any pointers?

[leadeater]

9 minutes ago, SteveiJobs said:

I feel like it might be related to the fact that the card is a server variant with no display outputs and is meant to be used for compute

Correct, if you cannot plug directly in to the GPU when doing VMs with passthrough then it cannot be used as the main display adapter. You can still use it for application GPU acceleration, Blender should use it so that's odd, but games will not be able to.

 

9 minutes ago, SteveiJobs said:

Horizon View

This is a very big application with quite a lot of setup and still isn't ideal for playing games sadly.

 

Unfortunately the best advice I can think of is to sell the GPU and get one that has display output, either from the Quadro/professional segment of Geforce. Not sure if you need the extra VRAM or not so I'll leave that up to you since I'm sure you know what you need.

[frazer_mc]

Try this: https://github.com/DualCoder/vgpu_unlock

Or https://forum.level1techs.com/t/gaming-on-my-tesla-more-likely-than-you-think/171185

 

I attempted vgpu passthrough this week, extremely hit or miss but the best community to ask is https://discord.gg/tCxUJEV275

 

There's mentions of a dummy x driver then configurating the nvidia card to use that dummy driver,

For Windows VMs you'll need vGPU support which is only for GRID licensed cards - ask on the discord I linked

 

But you can 100% do what you are describing - either through what I have linked above or maybe https://looking-glass.io/.

It takes time, you may find you can't do it but its all software limitations when it comes to Nvidia - you can always work around those

 

Hope this helps bro

[leadeater]

1 minute ago, frazer_mc said:

Try this: https://github.com/DualCoder/vgpu_unlock

Or https://forum.level1techs.com/t/gaming-on-my-tesla-more-likely-than-you-think/171185

 

I attempted vgpu passthrough this week, extremely hit or miss but the best community to ask is https://discord.gg/tCxUJEV275

 

There's mentions of a dummy x driver then configurating the nvidia card to use that dummy driver,

For Windows VMs you'll need vGPU support which is only for GRID licensed cards - ask on the discord I linked

 

Hope this helps bro

That's not the issue, Nvidia removed GPU passthrough restriction on consumer/Geforce GPUs and the RTX 6000 is a professional product and those have always supported it.

 

Without being able to plug a monitor directly in to the GPU it cannot bed used for anything other than application acceleration when passed through. Without passing it through then you need to use VMware Horizon or equivalent from other hypervisor vendors and also purchase Nvidia virtual GPU licenses.

 

VMware Horizon is really great though, we have a few thousand concurrent session setup at work and it even gets used for video editing with software like Adobe Premier and Aftereffects.

[frazer_mc]

Just now, leadeater said:

That's not the issue, Nvidia removed GPU passthrough restriction on consumer/Geforce GPUs and the RTX 6000 is a professional product and those have always supported it.

 

Without being able to plug a monitor directly in to the GPU it cannot bed used for anything other than application acceleration when passed through. Without passing it through then you need to use VMware Horizon or equivalent from other hypervisor vendors and also purchase Nvidia virtual GPU licenses.

 

VMware Horizon is really great though, we have a few thousand concurrent session setup at work and it even gets used for video editing with software like Adobe Premier and Aftereffects.

Maybe you're right, but maybe... https://looking-glass.io/

 

 I haven't tried it personally, but maybe this is OP's best bet?

[leadeater]

42 minutes ago, frazer_mc said:

Maybe you're right, but maybe... https://looking-glass.io/

 

 I haven't tried it personally, but maybe this is OP's best bet?

Forgot about that software, yes good idea, worth trying.

[frazer_mc]

Update: @SteveiJobs  I saw these - let us know if you managed to fix your problem

 

 

[SteveiJobs]

6 hours ago, frazer_mc said:

Update: @SteveiJobs  I saw these - let us know if you managed to fix your problem

 

 

I saw these too but I think I'm SOL because this particular card doesn't support driver switching. I also found a very hidden footnote on a product page implying that it required Nvidia vGPU for display output, meaning that it looks like I'm going to be paying for some extra licensing...

 

Oh well, thanks everyone!