2 Gamers, 1 CPU - Virtualized Gaming Build Log

[DaiGurenMK42]

Yes, unRAID is a paid-product and yes the virtualization technology we use (KVM) is open-source.  If you were so inclined, you could download a Linux distro, add all the software packages necessary for virtualization, and have your way at it.  The problem is there are lots of quirks to the kernel / distro / packages required to increase your chance of success with GPU pass through like this.  You also have to resort to using general purpose VM management tools that are a bit more complicated and cumbersome.

 

unRAID offers users the capabilities of a NAS, a virtualization host, and an application server all in the same package.  Our software manages storage devices in a unique way from standard software-defined arrays.  We use btrfs for our cache pool so that you can dynamically increase your storage over time by just adding more disks (as opposed to pre-defined RAID groups that cannot be modified once defined).  For our array storage, we use a parity-projected array with a union file system.  This means that there is no striping of data between multiple array disks.  This is why we offer the cache pool, because write performance after two devices in the array will drop a bit (30-50MB/s range).  The benefit is that if you lose more than 1 device, you don't actually lose all your data (like you would with a RAID 5).  Instead, you only lose the data on the devices you lost.  We provide fault-tolerance already for a single device (single parity), but we are adding dual-parity right now for an upcoming release.  The union file system makes it so that when browsing over the network, you just drop files into a "share" which could technically spread across multiple disks, but you'll see the aggregate view of all the data in that "share" for all the disks that participate with it.  Then if you were to look at any individual disk, you could still see and read individual files off those disks as well (because we don't stripe data).

 

We are trying to make all of this really cool technology much more accessible to a wider audience.  So while RYO (roll-your-own) NAS is definitely an option, if you want to get past the headache and jump right to the good stuff, our software cuts those corners for you.

 

Some people want to do it themselves others what to simplify it and still get all the features. I appreciate lime tech for thinking on the issue and developing a proper robust method for it to be applied. It's really not to complicated and it seems to support various hardware which is what I had trouble with trying to get steam on linuxmint working. Nothing I tried short of editing the files worked to get nvidia gpu working, had to go back to windows grudgingly. 

[limetechjon]

Some people want to do it themselves others what to simplify it and still get all the features. I appreciate lime tech for thinking on the issue and developing a proper robust method for it to be applied. It's really not to complicated and it seems to support various hardware which is what I had trouble with trying to get steam on linuxmint working. Nothing I tried short of editing the files worked to get nvidia gpu working, had to go back to windows grudgingly. 

 

Thanks for the kind words!  We are working hard to make this even easier for folks as time goes on.  We believe that the do-it-yourself PC and NAS market could really benefit from unRAID OS.  We make storage easy to protect and scale and we make isolation technologies like Docker containers and Virtual Machines much easier to use and accessible to the general techie.  Many of the competition out there is focused on big business/enterprise-class solutions.  While these offerings have their place, they can be extremely cumbersome to get started with and certainly aren't intuitive to use.  With our solution, there are far less steps involved and things are just much easier to use.

[leadeater]

That sounds expensive.

 

I'd prefer doing it manually in linux though, instead of unRaid. Because it would teach me so much more about how the system works, and I'd get a much better understanding of what I need to do when something goes wrong.

 

Also, the main reason that I would consider doing this is because I might want to play DX12 games, but I do NOT want to install win 10 directly on to my system. With this I can have a virtual Win 10 system while running some linux distro as the main OS. I really really want to get myself a second rig to set it up on, because I don't expect it'll be a quick process to get everything working.

 

It would be so awesome to have a Bedrock linux system running with ubuntu/arch/gentoo system stacks, and a gimped win 10 on top of that with a separate GPU.  

 

Yea the big extra cost is in the Windows licensing, not having to run a dedicated GPU per VM would help offset the cost though. I wouldn't seriously expect anyone to be actually using that in a home setup, was just interested in RemoteFX and at the time there were very few things that could visualize a GPU and keep DX support which was also able to render a game at playable fps. Things you do because it fun, but not sensible  .

 

It worked well enough I was thinking about using one of my unused servers as a LAN party in a box. Plug it in and anyone with a Windows 8+ (Win 7 Ent/Ult also) device can connect in using RDP with RemoteFX, all games pre-installed in the VM so no messing around trying to organise who has what running which mods etc etc. Never got round to it, likely never will.

[Loadent]

Question on the capabilitys of the virtual machine. I haven't set one up in years. Linus used a 5960X, will a 5930K have enough horse power? And using the 5930K, can I use 2 Gtx 980 and 2 Gtx 970 and run both in SLI?

Edit - do you have to purchase additional win 10 licenses for each VM?

[leadeater]

Question on the capabilitys of the virtual machine. I haven't set one up in years. Linus used a 5960X, will a 5930K have enough horse power? And using the 5930K, can I use 2 Gtx 980 and 2 Gtx 970 and run both in SLI?

Edit - do you have to purchase additional win 10 licenses for each VM?

 

Yes each VM would need a license. As for SLI it should work as your passing through the hardware device directly to the VM but there is no real way of knowing for sure without try it. There might be some issue due to the visualization that could cause problems like stuttering.

 

The 5930k would work just fine, you could go to lower model than that also as long as it supports VT-d, but it all depends on the game. What a game needs to use and can use varies wildly, 'most' games released 2014 and earlier don't see any big gains past 3 cores and run fine on 2.

[Dabombinable]

So.....I could make use of the GTX 650ti in my main rig as more than a PhysX card when using Windows......

[Frillite]

I wish I could afford all the hardware mentioned.

 

[taavi]

 

 

 

This is also possible using Windows Server Hyper-V with RemoteFX. Software cost is higher but hardware cost can be lower as you can share the GPU and CPU cycles rather than having to split them between VMs. It's also much easier to setup and the mentioned USB provisos with unRAID are not an issue.

 

I had a good tinker with this back in Server 2008 R2 with an AMD 6970 running 2 Windows 7 VMs playing NFS Shift 2. Used this game as it would show up input issues really fast. Worked great with both games running 60~ fps from memory, was while ago so testing me memory. Windows Server 2012 R2 does a better job now too, was actually thinking about setting up again.

 

Interesting video though, loved seeing a similar goal achieved in a different way.

 

 

Hyper-V server and RemoteFX are both free - how come the software cost is higher?

[leadeater]

Hyper-V server and RemoteFX are both free - how come the software cost is higher?

 

I was more thinking of the Hyper-V role in Windows Server. Wasn't fully aware that Microsoft released an actual Windows Server Hyper-V edition that was free, thanks for the heads up on that one. I deal mostly with VMware for my day job so I'm not a huge expert in the goings on with Hyper-V and my home servers are also VMware.

 

Guess the only down side to the RemoteFX setup would be the requirement of client devices compared to unRAID which you plug directly in to, could also be a positive too.

[Guest]

It's nice to see some Linux content and software used on the LTT channel. I'm surprised you used something as dated as unRaid though, being based on Slackware with Kernel 2.6. 

[Guest]

-

UnRaid is based on slackware, unRaid is Linux. Just with some proprietary dated crap. 

[iain]

Nice video, but a few things are bothering me about it.

 

• Don't pay for the Lime Tech Unraid server, just get the FREE copy of KVM. No point in paying for something that's free (Unraid is just running KVM).
• I noticed in the video you have changed your DNS to Googles DNS... why? No point in doing that on end devices, I then noticed that you had to use the server IP to connect to the web console. The reason you had to do this was because it's now not contacting your local DNS server (aka your Router), it's now contacting Googles DNS.

A question for you Linus, did you have any Input lag at all? If the motherboard has split USB controllers you can pass through each of them and get 0 input lag.

Yes you can grab a copy of KVM and set it up on Linux. But with pass though this is no easy task. The limetech solution saves the user a lot of time and headache. 

[Altecice]

Yes you can grab a copy of KVM and set it up on Linux. But with pass though this is no easy task. The limetech solution saves the user a lot of time and headache. 

 

Its exactly the same method - http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

You can also achieve the same thing in KVM in GUI as well.

[mikat]

Ever thought to yourself - My computer is so powerful that it could conceivably run TWO copies of this game.. How would I do that? Well, here's the answer, friends!

 

Vessel: https://www.vessel.com/videos/V0Gjb9_OL

YouTube: TBD

 

 

Cooler Master Master Case 5

Amazon: http://geni.us/49pd

NCIX: http://bit.ly/1K3CVzF

 

Cooler Master V850 Power Supply

Amazon: http://geni.us/3Vb7

NCIX: http://bit.ly/1GDRMAM

 

Intel 5960X CPU

Amazon: http://geni.us/1nya

NCIX: http://bit.ly/1fNNdwA

 

ASUS X99 Deluxe

Amazon: http://geni.us/14q3

NCIX: http://bit.ly/1G8grT7

 

Nvidia Titan X

Amazon: http://geni.us/3AQ2

NCIX: http://bit.ly/1BNzb4D

 

NVIDIA GTX 980 Ti

Amazon: http://geni.us/lDd

NCIX: http://bit.ly/1LJa9vK

 

Intel 730 Series SSD

Amazon: http://geni.us/3f27

NCIX: http://bit.ly/1jqcGxT

 

Corsair Vengeance LPX DDR4 RAM

Amazon: http://geni.us/2y54

NCIX: http://bit.ly/1Qkp8tK

 

Seagate Barracuda HDD

Amazon: http://geni.us/2kBD

NCIX: http://bit.ly/1PpnRn0

 

Lime Technology unRAID Server Pro: http://lime-technology.com/

you've been featured on their website linus

http://lime-technology.com/unraid-featured-on-linustechtips/

[iain]

Its exactly the same method - http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

You can also achieve the same thing in KVM in GUI as well.

Yes but that is not simple to setup! Thats my point. If you could get all of that working in under 2 hours with no prior knowledge I would be amazed. 

[limetechjon]

Question on the capabilitys of the virtual machine. I haven't set one up in years. Linus used a 5960X, will a 5930K have enough horse power? And using the 5930K, can I use 2 Gtx 980 and 2 Gtx 970 and run both in SLI?

Edit - do you have to purchase additional win 10 licenses for each VM?

The 5930X has 6 cores (12 threads), so while it's not quite as powerful as the one Linus used (which had 16 threads), it should still be more than enough.  I did a blog on our website where I compared VMs with the same GPUs and different vCPU assignments.  3D Mark benchmarks and all.  Link:  http://lime-technology.com/gaming-on-a-nas-you-better-believe-it/

[limetechjon]

Yes each VM would need a license. As for SLI it should work as your passing through the hardware device directly to the VM but there is no real way of knowing for sure without try it. There might be some issue due to the visualization that could cause problems like stuttering.

 

The 5930k would work just fine, you could go to lower model than that also as long as it supports VT-d, but it all depends on the game. What a game needs to use and can use varies wildly, 'most' games released 2014 and earlier don't see any big gains past 3 cores and run fine on 2.

 

We haven't personally tested SLI in a while.  The first time we tried it, it did not work.  Yes, both the GPUs can show up in the VM at the same time, but the option in NVIDIA to enable SLI wasn't showing up.  From what we've researched, this probably doesn't work, although there were a few people at one point that claimed they succeeded, but never posted video to prove it, so I'm a little skeptical.

 

That said, we do believe that Crossfire should work with AMD-based graphics devices, but again, haven't personally tested this yet.  There are some technical differences in the implementation of SLI vs. Crossfire that leads us to believe this.  Generally speaking, don't count on this working for right now.

[limetechjon]

Hyper-V server and RemoteFX are both free - how come the software cost is higher?

 

Have you ever tried doing it with a GTX-series GPU?  Doesn't work from what I understand.  You have to buy the expensive Quadro cards.

[limetechjon]

It's nice to see some Linux content and software used on the LTT channel. I'm surprised you used something as dated as unRaid though, being based on Slackware with Kernel 2.6. 

 

What are you talking about?!  unRAID 6 was just released in July of this year and is based on Linux Kernel 4.1 right now.  We're as modern as it gets!

[Guest]

What are you talking about?!  unRAID 6 was just released in July of this year and is based on Linux Kernel 4.1 right now.  We're as modern as it gets!

Really need to update your FAQ then. 

 

"Motherboard and components compatible with unRAID software (based on Slackware), and Linux kernel 2.6.*" -- Source: Your FAQ

http://lime-technology.com/wiki/index.php/FAQ

[limetechjon]

Its exactly the same method - http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM

You can also achieve the same thing in KVM in GUI as well.

 

This is not the same method at all.  We spent months and months reviewing the processes for getting this to work properly and I can tell you the traditional Linux distro method is extremely complicated and time-consuming.  What distro do you pick?  What software packages do you need to install?  How do you configure GRUB properly?  How do you configure the kernel properly?  How about storage?  How do you expand an existing RAID group once you define it?  And what about ACS overrides for IOMMU groups?

 

The difference with traditional distros vs. unRAID is that traditional distros don't see it as their job to provide virtualization as a core feature.  They view it as something enabled in the kernel that specific software packages can take advantage of, but its not a core OS feature.  This means you have to literally download and install all the features and components for it yourself.  With unRAID, we prebuilt our OS to already deliver all of that functionality and out of a single management interface.  We as the vendor are taking the responsibility to providing those features directly, as part of our distribution.

 

Virtualization is a powerful tool, and open source Linux solutions have advanced much in the past year, but in terms of ease of use, it's still a long-stretch to hand someone virt-manager and expect them to figure out how to get GPU pass through to a VM working or for them to figure out how to manage storage pools in libvirt.  Our goal is to bring this technology within reach of the prosumer audience of technology enthusiasts (DIY PC guys) who know enough to build their own rigs, but aren't necessarily Linux / virtualization experts.

[limetechjon]

Really need to update your FAQ then. 

 

"Motherboard and components compatible with unRAID software (based on Slackware), and Linux kernel 2.6.*" -- Source: Your FAQ

http://lime-technology.com/wiki/index.php/FAQ

 

Right at the top of that page in big red text:  Important Note: this FAQ has not been updated for unRAID v6, so some information is out-of-date!

[leadeater]

Have you ever tried doing it with a GTX-series GPU?  Doesn't work from what I understand.  You have to buy the expensive Quaddro cards.

 

At the time I only had an NVidia 8800 GTS which doesn't meet the DirectX requirements. The only hardware requirements are DirectX 11+ and WDDM 1.2+ drivers. The supported hardware lists only show the workstation/server variants from AMD and NVidia but the consumer cards that meet those requirements like the 6970 I used should work. If I had a more recent NVidia card I'd check. NVidia do tend to be more aggressive with their product restrictions so I wouldn't be surprised if it didn't work on non Quaddro cards. 

 

Having revisited the Hyper-V and RemoteFX requirements I have found a show stopper though. For the VM the only supported Windows edition is Enterprise for Windows 7 & 8, can't find info on Windows 10 but assuming the same no home user or small business could use Hyper-V and RemoteFX. Enterprise editions are only available with a Microsoft volume license agreement.

[limetechjon]

At the time I only had an NVidia 8800 GTS which doesn't meet the DirectX requirements. The only hardware requirements are DirectX 11+ and WDDM 1.2+ drivers. The supported hardware lists only show the workstation/server variants from AMD and NVidia but the consumer cards that meet those requirements like the 6970 I used should work. If I had a more recent NVidia card I'd check. NVidia do tend to be more aggressive with their product restrictions so I wouldn't be surprised if it didn't work on non Quaddro cards. 

 

Having revisited the Hyper-V and RemoteFX requirements I have found a show stopper though. For the VM the only supported Windows edition is Enterprise for Windows 7 & 8, can't find info on Windows 10 but assuming the same no home user or small business could use Hyper-V and RemoteFX. Enterprise editions are only available with a Microsoft volume license agreement.

 

A few other issues with Hyper-V / RemoteFX.  First, RemoteFX is not designed so that you can plug a monitor into the back of the GPU and use it like a normal desktop.  Instead, you'd have to have a completely separate device as a client (laptop, desktop, tablet, etc.) and then use Microsoft RDP to connect to it.  And even then, like you noted, you would need to have Windows Enterprise to do it, which means volume license agreement and big $$$.  And at the end of the day, you're still getting a remote graphics connection, not a local graphics connection like with unRAID.

 

Another issue with RemoteFX is that it will require you to use professional grade GPUs.  The consumer grade devices will not work properly.  The reason is that RemoteFX is designed to take advantage of things like NVIDIA GRID technology which allows you to share a single GPU among multiple VMs concurrently.  That said, it's not designed for gaming use-cases, but rather, other 3D graphics / modeling software (think CAD applications).  If you tried to do multi-user gaming on a NVIDIA GRID GPU, I don't think any self-respecting gamer would appreciate the performance because you will be taxing that single GPU much harder.

 

I would challenge anyone to try and use VirtualBox, VMWare, or Hyper-V to try and recreate what Linus did in this video using the same hardware.  We have found that other solutions are far more complicated to set up and don't work with nearly as many different GPUs / motherboards as ours does.

[Dabombinable]

This is not the same method at all.  We spent months and months reviewing the processes for getting this to work properly and I can tell you the traditional Linux distro method is extremely complicated and time-consuming.  What distro do you pick?  What software packages do you need to install?  How do you configure GRUB properly?  How do you configure the kernel properly?  How about storage?  How do you expand an existing RAID group once you define it?  And what about ACS overrides for IOMMU groups?

 

The difference with traditional distros vs. unRAID is that traditional distros don't see it as their job to provide virtualization as a core feature.  They view it as something enabled in the kernel that specific software packages can take advantage of, but its not a core OS feature.  This means you have to literally download and install all the features and components for it yourself.  With unRAID, we prebuilt our OS to already deliver all of that functionality and out of a single management interface.  We as the vendor are taking the responsibility to providing those features directly, as part of our distribution.

 

Virtualization is a powerful tool, and open source Linux solutions have advanced much in the past year, but in terms of ease of use, it's still a long-stretch to hand someone virt-manager and expect them to figure out how to get GPU pass through to a VM working or for them to figure out how to manage storage pools in libvirt.  Our goal is to bring this technology within reach of the prosumer audience of technology enthusiasts (DIY PC guys) who know enough to build their own rigs, but aren't necessarily Linux / virtualization experts.

And that is all I need to ditch Windows as my main OS. I spent most of my spare time last month trying to get GPU pass through working, first with my GTX 650ti for the distro and GTX 970 for the VM, then with the Intel HD graphics for the distro to make things a bit easier. Needless to say it didn't pan out that well.