NVIDIA to officially allow GPU passthrough to Windows VM from Linux host

[gjsman]

Summary

Today, NVIDIA announced that they will officially support the ability to have a Windows VM on a Linux host and passthrough a GPU to it. Although this was technically possible earlier, NVIDIA made it very difficult with Error 43 on their consumer hardware, requiring gamers to try preventing NVIDIA's drivers from knowing they were running inside a VM. With this change, running a Linux operating system as your daily driver and having a Windows VM for certain games is now easily possible.

 

However, NVIDIA can't do anything without a catch. Because this is not SR-IOV, you will need 2 GPUs: 1 for Linux, and 1 being passed through to Windows. Which, considering the GPU shortage, means this is a shallow victory.

 

Quotes

Quote

NVIDIA has enabled GPU passthrough beta support for a Windows virtual machine on GeForce GPUs. What does this mean?
With virtualization enabled, GeForce customers on a Linux host PC can now enable GeForce GPU passthrough on a virtual Windows guest OS.  There are a few GeForce use cases where this functionality is beneficial such as:

• GeForce customers wanting to run a Linux host and be able to launch a Windows virtual machine (VM) to play games 
• Game developers wanting to test code in both Windows and Linux on one machine

 

What class of virtualization is supported on GeForce GPUs?
GeForce GPU passthrough supports 1 virtual machine.  SR-IOV is not supported on GeForce.  If you want to enable multiple virtual machines to have direct access to a single GPU or want the GPU to be able to assign virtual functions to multiple virtual machines, you will need to use NVIDIA Tesla, Quadro, or RTX enterprise GPUs.

 

Which driver is GeForce virtualization (beta) supported on? 
GeForce virtualization (beta) is supported on R465 or higher drivers.

 

Which GeForce GPUs and Windows OSes support virtualization?
The feature is enabled on all GeForce/TITAN GPUs supported in the R465 driver (Kepler and later for Desktop; Maxwell and later for Notebook) for Windows 10.

 

Do you need to have more than one GPU installed or can you leverage the same GPU being used by the host OS for virtualization?
One GPU is required for the Linux host OS and one GPU is required for the Windows virtual machine.

 

My thoughts

I'm happy, but at the same time, a bit miffed that this isn't SR-IOV with the ability to split a GPU into parts (like you can with CPU cores), so the requirement for 2 GPUs is quite unfortunate.

 

Sources:

https://nvidia.custhelp.com/app/answers/detail/a_id/5173/~/geforce-gpu-passthrough-for-windows-virtual-machine-%28beta%29

[Windows7ge]

Does this mean they're removing the block from the driver or only new GPUs will have pass-through ability?

 

If the latter I still don't plan on buying an NVIDIA GPU.

[Granular]

Great for those situations when you want to use an open source OS, but don't want open source graphics drivers, so you run Nvidia and also don't really want an open source OS, so you run Windows in a VM.

[Kisai]

18 minutes ago, gjsman said:

Summary

Today, NVIDIA announced that they will officially support the ability to have a Windows VM on a Linux host and passthrough a GPU to it. Although this was technically possible earlier, NVIDIA made it very difficult with Error 43 on their consumer hardware, requiring gamers to try preventing NVIDIA's drivers from knowing they were running inside a VM. With this change, running a Linux operating system as your daily driver and having a Windows VM for certain games is now easily possible.

 

However, NVIDIA can't do anything without a catch. Because this is not SR-IOV, you will need 2 GPUs: 1 for Linux, and 1 being passed through to Windows. Which, considering the GPU shortage, means this is a shallow victory.

 

Quotes

 

My thoughts

I'm happy, but at the same time, a bit miffed that this isn't SR-IOV with the ability to split a GPU into parts (like you can with CPU cores), so the requirement for 2 GPUs is quite unfortunate.

 

Sources:

https://nvidia.custhelp.com/app/answers/detail/a_id/5173/~/geforce-gpu-passthrough-for-windows-virtual-machine-%28beta%29

Well, you could always use the iGPU on Intel systems to do this, that's how you did it on Windows as well. So this isn't really anything new in that regard.

 

Though I don't see the point, this will just enable hacking of MMO games by using the virtual machine so the client OS isn't aware of what's going on. So I don't expect competitive MMO games to permit their games to be played inside a Linux hosted virtualized environment any more than they currently permit this already on a MacOS hosted or Windows hosted one.

 

If you are playing games as your primary pc purpose, you run Windows. If you use your Linux for some other reason and occasionally play games on it, then maybe this gives you an option that wasn't previously available. However, again, this falls back to game developers to not purposely break the game running inside a VM. Many games have broken mouse movement inside a VM because of mouse cursor translation not matching the the host os.

 

 

[Bombastinator]

O.O
I avoided this in the past because it was such a PITA.  My memory was it was the only way to have a really secure windows system.  My work around is currently using my PC as more or less nothing but a games console and just not doing anything vaguely of consequence on it.  This could make things viable though.

[GoodBytes]

Keep in mind that if you are doing development under Linux, Nvidia offers drivers for using your current Nvidia GPU (GeForce and Quadro) for CUDA acceleration under WSL2

https://developer.nvidia.com/cuda/wsl/download

 

So, I don't think it is the end of the world the current limitation, unless you don't have Intel integrated graphics, and you really want to play games under Linux through a VM running under Windows for some reason... I guess TuxRacer is that of a compelling game....

[leadeater]

9 hours ago, gjsman said:

you will need 2 GPUs: 1 for Linux, and 1 being passed through to Windows

Well up until Ryzen got very popular the majority had an Intel iGPU anyway, I would say most still do.

[Master Disaster]

3 hours ago, leadeater said:

Well up until Ryzen got very popular the majority had an Intel iGPU anyway, I would say most still do.

Plus Linux will run on pretty much anything (as long as you go for a low weight DM like XFCE), you could just go buy a GT710 and use it to run Linux then pass your RTX through to Windows.

[Bombastinator]

3 hours ago, Master Disaster said:

Plus Linux will run on pretty much anything (as long as you go for a low weight DM like XFCE), you could just go buy a GT710 and use it to run Linux then pass your RTX through to Windows.

So not actually much simpler to set up?  That’s unfortunate.

[Arika]

What's the current performance penalty of running Windows in such a way? Last time i tried to run a VM (though it was a Linux VM within Windows) it was not a great experience.

 

Since it's a dedicated GPU being passed through, one would assume there should be no GPU penalty?

[Bombastinator]

1 minute ago, Arika S said:

What's the current performance penalty of running Windows in such a way? Last time i tried to run a VM (though it was a Linux VM within Windows) it was not a great experience.

 

Since it's a dedicated GPU being passed through, one would assume there should be no GPU penalty?

My memory was someone claimed 10%.  I don’t know how accurate that is though.

[igormp]

That's really nice, especially if you have an integrated graphics to use on your host system.

23 hours ago, gjsman said:

I'm happy, but at the same time, a bit miffed that this isn't SR-IOV with the ability to split a GPU into parts (like you can with CPU cores), so the requirement for 2 GPUs is quite unfortunate.

Well, that's not possible to solve since the geforce cards (and most quadro/teslas) lack actual hardware that's required to have proper SR-IOV.

 

13 hours ago, GoodBytes said:

Keep in mind that if you are doing development under Linux, Nvidia offers drivers for using your current Nvidia GPU (GeForce and Quadro) for CUDA acceleration under WSL2

It's slower than native, and even worse to get stuff working than your regular CUDA install (which is a PITA by itself already).

[Master Disaster]

21 hours ago, Bombastinator said:

So not actually much simpler to set up?  That’s unfortunate.

The same thing is true of any VM Hardware Passthrough. Devices connected to your computer can only talk to one bus at a time, if you want to pass them through to a VM then it will always disable them on the host, that's just the nature of virtualised hardware.

 

Its not like you can buy a 6700XT (April Fools joke right there), stick it in a computer and magically use it in both a Windows host and Linux VM at the same time (or vice versa).

[AlexGoesHigh]

21 hours ago, Arika S said:

What's the current performance penalty of running Windows in such a way? Last time i tried to run a VM (though it was a Linux VM within Windows) it was not a great experience.

 

Since it's a dedicated GPU being passed through, one would assume there should be no GPU penalty?

VM being clunky is a problem of lacking graphics, since VM emulate hardware, graphics are emulated and run on the host CPU, that's why running anything with a GUI on a VM feels clunky nevermind that anything that needs graphics horsepower would be a slideshow if it manages to run at all, non graphical workloads tend to be 10-20% slower that on bare metal depending on the workload itself.

 

Once you give a full real GPU to a VM, performance its basically the same as bare metal unless the program gets CPU bottlenecked which the VM performance penalty applies.

[OrcephRye]

6 hours ago, AlexGoesHigh said:

VM being clunky is a problem of lacking graphics, since VM emulate hardware, graphics are emulated and run on the host CPU, that's why running anything with a GUI on a VM feels clunky nevermind that anything that needs graphics horsepower would be a slideshow if it manages to run at all, non graphical workloads tend to be 10-20% slower that on bare metal depending on the workload itself.

 

Once you give a full real GPU to a VM, performance its basically the same as bare metal unless the program gets CPU bottlenecked which the VM performance penalty applies.

This is not exactly true. If a person was to put a lot of work into their VM setup then you can get very very close to bare metal performance but that just does't happen simply because you pass a GPU through. 

 

Also performance hit from using a VM varies widely dependent on the application. 1 game may get near perfect native performance while another takes a 10% hit. The best way to describe it is you are gonna get a small latency increase that effects the behavior of the whole system. You can measure this with different latency tools for Windows but everything memory reads/writes to I/O in general is gonna be effected. With the right tweaks the impact can be small enough to not matter but those tweaks are not on by default and often depend on the hardware you use. 

 

The biggest hit is in I/O to disk. Even if you passthrough a controller or a M.2 drive there is still a noticeable hit under certain I/O loads that just isn't going away. (Although if gaming is your concern passing a M.2 drive or a SATA controller through to the VM basically makes it unnoticeable unless you run a benchmark of some sort)

 

The list can go on. If you are doing VR better make sure you pass though a USB controller (or hope your video card has that nifty USBC port) because without it the latency of the emulated USB is too much. Getting high polling rate mice to work well takes tweaking as well. For the love of all that is holly don't use e1000 drivers for your virtual nic. Know your CPU core layout. Isolate cores. Pray to the VM gods by sacrificing a virgin CPU. Do the hooky pooky... cuz.. really that is what it is all about? 

But seriously it can be a lot of work and really rewarding if you are into tweaking software/hardware for the best performance. 

 

Also to stay on topic I do not get what Nvidia is doing here. What is the advantage of adding this to the driver? Making it a tiny bit easier to isolate the GPU for passthrough? Is doing it 'Nvidia way' suppose to yield better performance somehow? What is the point!? 

[SFFDesigns]

On 3/30/2021 at 2:10 PM, Windows7ge said:

Does this mean they're removing the block from the driver or only new GPUs will have pass-through ability?

 

Removed from driver, so all GPUs 600 series and up I believe.

[igormp]

7 minutes ago, OrcephRye said:

Also to stay on topic I do not get what Nvidia is doing here. What is the advantage of adding this to the driver? Making it a tiny bit easier to isolate the GPU for passthrough? Is doing it 'Nvidia way' suppose to yield better performance somehow? What is the point!? 

Make our lives easier without having to do any workaround to get their GPUs working under passthrough scenarios.

[cvbx]

On 3/31/2021 at 1:13 PM, Arika S said:

What's the current performance penalty of running Windows in such a way? Last time i tried to run a VM (though it was a Linux VM within Windows) it was not a great experience.

 

Since it's a dedicated GPU being passed through, one would assume there should be no GPU penalty?

My experience is around 5%, depending on the type of workload and tweaks to the VM (cpu pinning, numa, hugetables...) I am more than willing to accept that tradeoff, since hardware these days is overpowered anyway and I wouldnt notice the difference from bare metal even if I tried. But keep in mind, virtualization with gpu/pcie-passthrough is still not a one click solution and performance depends on many factors, so your results may vary.

 

Regarding the topic at hand: i very much appreciate the efforts that go into opening hardware to virtualization by the common user. More and more features become available outside of enterprise environments, and thats a good thing. Currently i use AMD only, just because it was simpler to get stuff working in a VM. News like this one might change that.

[Bitter]

How about the other way around or from Windows host to Windows VM? I want to give sketchy software direct access to my GPU in Linux or Windows running on VM!