7 Gamers, 1 CPU - Ultimate Virtualized Gaming Build Log

[Neoistheone2000]

These could potentially be used for a Internet cafe.

Just remember kids, these video cards support Eyefinity too.

I want to see that, put 3 monitors per each card and give me a 21 Monitors on 7 VMs

[3dfx]

Better off playing on that machine solo.

[danial9]

any one know who is that girl at 10.38 in the video? 

last time i checked he had no female employee.

[Marinatall_Ironside]

any one know who is that girl at 10.38 in the video? 

last time i checked he had no female employee.

His wife Yvonne?

[kmgolf08]

This setup is epic!!  Clearly unRAID is one heck of a system.

 

Does anyone know what that keyboard (brand/model) Linus used during the game play at the end of the video is? Love the old school look.

[TheSLSAMG]

Wouldn't work.

 

The rear I/O is two slots in width and the PCBs are too long for Linus to get the cards out ever again.

You could use K|NGP|Ns and pull the clips off of the PCI-E slots, but that's really expensive.

[byalexandr]

You could use K|NGP|Ns and pull the clips off of the PCI-E slots, but that's really expensive.

I actually had an idea of using PCI-E riser cables, spacing out the GPUs, and then just getting a custom case from ProtoCase to fit the bill.

[52ObiB]

I'm glad to hear you saying that.  We are just about to start testing with NVMe and we were worried about using btrfs due to this spec sheet from Intel (scroll to the bottom about File System Recommendations):  

 

Curious how performance holds up over time and how you configured TRIM given that they don't want the filesystem to manage discard.  Also curious if you just threw btrfs on there on a whim and it's just been working out or if you know something we don't about the safety of using btrfs on NVMe devices.

 

I'm familiar with that document. I might not know anything you don't. An issue I encountered was nasty firmware bug related to ACPI sleep resuming that is much better in newer firmware. 

 

btrfs mounts by default don't use the discard option which I understand is the important part of their recommendation (not exclusive to ext4 and xfs but perhaps they've only tested with those).

 

I'm actually using btrfs (without discard) -> dm-crypt -> /dev/nvme*n1 which then is completely different in appearance to the NVMe drive.

[limetechjon]

I'm familiar with that document. I might not know anything you don't. An issue I encountered was nasty firmware bug related to ACPI sleep resuming that is much better in newer firmware. 

 

btrfs mounts by default don't use the discard option which I understand is the important part of their recommendation (not exclusive to ext4 and xfs but perhaps they've only tested with those).

 

I'm actually using btrfs (without discard) -> dm-crypt -> /dev/nvme*n1 which then is completely different in appearance to the NVMe drive.

 

Interesting feedback.  Thanks for that.  Encryption is another feature on the roadmap for future inclusion, so good to know that's a working setup for you.

[igormp]

What speed are the PCI-e slots running? Each Xeon provides 40 lanes (for a total of 80 lanes), that means that each card is running at 8x?

[Marinatall_Ironside]

What speed are the PCI-e slots running? Each Xeon provides 40 lanes (for a total of 80 lanes), that means that each card is running at 8x?

Top slot runs at 16x bandwidth while the rest is at 8x bandwidth.

[melias24]

Here i am sitting viewing my amazon cart, at $22,407.94 for just the PC excluding all the watercooling, case and other stuff. I really don't know how it is possible. Oh well, i feel bad for Linus for having to be the one to pjut that all together i'd be nervous ASF

[vagabond139]

Maan!! Whoow!

 

@LinusTech I expected a really long video of this one. Especially considering that it is a $30k PC.

 

You could have asked Super Flower for their 2000w PSU and throw in 7 x 980 Ti in there.

Nah he should have got two of them in a Caselab TX10 with 7 Fury X's and get some new breaker(s) installed  to handle that much wattage.

[Yaerox]

Did I got this right. Your PC is basically running the Lime Server OS. On Lime you simply start running 7 VMs and redirect all the hardware sources the way you want to those VMs?

 

I'm a little bit confused because Lime seems to be really good, but never heared about it. Can you give us some more detailed informations about this Virtualisation part? How does all those KEyboards/Mouses are getting connected? USB-Hubs? Did you configured one graphics card fix for each VM, so that you never have to change HDMI cables from one to another graphics card?

 

Regards.

No need to say insane build I guess you already knew long time before

[zipeldiablo]

From what I've read, the issue is nVidia actively blocking their driver from starting if it detects it's in a virtual machine, not an issue with ESXi itself...

That's why recent versions of qemu allow you to hide the kvm state. But yeah they are actively blocking us, so for now it is close to impossible to passthrough recent gpu in xen for example.

[limetechjon]

Did I got this right. Your PC is basically running the Lime Server OS. On Lime you simply start running 7 VMs and redirect all the hardware sources the way you want to those VMs?

 

I'm a little bit confused because Lime seems to be really good, but never heared about it. Can you give us some more detailed informations about this Virtualisation part? How does all those KEyboards/Mouses are getting connected? USB-Hubs? Did you configured one graphics card fix for each VM, so that you never have to change HDMI cables from one to another graphics card?

 

Regards.

No need to say insane build I guess you already knew long time before

 

There is a lot of information available about unRAID, the OS used in the build, on our website at http://lime-technology.com.  unRAID boots from a USB flash stick where the OS lives, and then all your storage devices are managed by unRAID.  You can create a pool of disks for high-performance or an array of disks for high-capacity.  Either way, data is protected.  The pool can be used to act as a write-cache for the array, giving you faster write performance in real-time, then moving those files to the array at night, when you're not using the system anyway.  The pool also can be used to store your virtual disk images for your virtual machines, giving them pretty decent performance, especially if you use SSDs for the pool.

 

The virtualization technology used in unRAID is QEMU/KVM.  However, unlike traditional Linux distributions, unRAID includes all the components required to do all of this as soon as you boot up.  In addition, it's all presented through an easy-to-use web interface.  As someone earlier in the thread mentioned, the virtualization technologies used for this are freely available to download, install, and configure on any platform, but they are not exactly easy-to-use for someone looking to do things like what Linus did in this video.

 

That's why recent versions of qemu allow you to hide the kvm state. But yeah they are actively blocking us, so for now it is close to impossible to passthrough recent gpu in xen for example.

 

Yup, and unRAID actually automatically applies tweaks to the VM configuration when an NVIDIA GPU is added so ensure no Code 43 errors.

[52ObiB]

 

Yup, and unRAID actually automatically applies tweaks to the VM configuration when an NVIDIA GPU is added so ensure no Code 43 errors.

 

Nice @limetechjon!

 

I think I'm familiar with those tweaks for QEMU (see the vfio mailing lists for tips on using them via libvirt and qemu options), but as @zipeldiablo mentions, it's reasonable to assume there is a risk of NVIDIA finding new ways to assert virtualization is in use to 'Code 43' with non > K1200 cards.

 

For me this means I use a K2200 for work related VMs to mitigate that risk besides other advantages the card has - low heat, low profile, different drivers.

 

That aside I might keep trying to use a GeForce card for games - which is not critical infrastructure - until NVIDIA stops doing that, which I hope they do soon.

 

Hopefully this comment helps someone who is targeting either games or work use.

[GeekPower0]

I would love to see this done with Nvidia's GRID GPU's

[maran1002]

Can you talk about the software and how did you virtualize?

[limetechjon]

Can you talk about the software and how did you virtualize?

 

You should check out the 2 Gamers 1 CPU video, he covers this step by step.

[aw_]

Can you talk about the software and how did you virtualize?

 

KVM with VFIO is the underlying virtualization technology, I've given a couple talks on the development of this over the past few years:

 

http://www.linux-kvm.org/images/e/ed/Kvm-forum-2013-VFIO-VGA.pdf

http://www.linux-kvm.org/images/b/b3/01x09b-VFIOandYou-small.pdf (https://www.youtube.com/watch?v=NhZ9eIpg2nM)

 

And a blog with several articles and a how-to guide:

http://vfio.blogspot.com

 

It would be insane if anything other than the UEFI/OVMF method is used on this system given the guest and new cards (2014 talk).  unRAID has a nice web GUI and adds some non-upstream patches that you hopefully won't need if you select your hardware correctly, but this can pretty easily be setup on a smaller scale with any modern Linux distribution and libvirt.  The hardware must support an IOMMU to make this work, like Intel VT-d.

[limetechjon]

KVM with VFIO is the underlying virtualization technology, I've given a couple talks on the development of this over the past few years:

 

http://www.linux-kvm.org/images/e/ed/Kvm-forum-2013-VFIO-VGA.pdf

http://www.linux-kvm.org/images/b/b3/01x09b-VFIOandYou-small.pdf (https://www.youtube.com/watch?v=NhZ9eIpg2nM)

 

And a blog with several articles and a how-to guide:

http://vfio.blogspot.com

 

It would be insane if anything other than the UEFI/OVMF method is used on this system given the guest and new cards (2014 talk).  unRAID has a nice web GUI and adds some non-upstream patches that you hopefully won't need if you select your hardware correctly, but this can pretty easily be setup on a smaller scale with any modern Linux distribution and libvirt.  The hardware must support an IOMMU to make this work, like Intel VT-d.

 

Just to confirm, all the VMs were running in OVMF instances (and with proper split VARS support).  The GPUs actually didn't seem to support UEFI natively, however, so I had to configure the VMs first using VNC for graphics, then add the GPUs as a secondary, install GPU drivers, then remove the VNC graphics.  While you don't see graphics output while Windows is booting, as soon as the drivers are loaded, the monitor lights up.  Pretty much standard procedure.

[zipeldiablo]

Yup, and unRAID actually automatically applies tweaks to the VM configuration when an NVIDIA GPU is added so ensure no Code 43 errors.

 

wat? Wish i knew that before spending two weeks installing my windows gaming vm on fedora, switching to xen + unraid o/ (IF it supports gtx980 ti)

You guys should do a tutorial like post to explain a bit how to set things up with unraid, lot of post to use qemu/kvm but not much on gaming with unraid

[euzen01]

Hust plane DOPE!

[limetechjon]

wat? Wish i knew that before spending two weeks installing my windows gaming vm on fedora, switching to xen + unraid o/ (IF it supports gtx980 ti)

You guys should do a tutorial like post to explain a bit how to set things up with unraid, lot of post to use qemu/kvm but not much on gaming with unraid

... There are multiple guides under the support -> videos section and Linus does a step by step for setup in the 2 gamers 1 CPU video.