7 Gamers, 1 CPU - Ultimate Virtualized Gaming Build Log

[unijab]

3DMark is down for me right now, but when it comes back compare the M4000 to the R9 Nano, I checked before posting and found reports of 16xxx for the M4000 vs 14xxx for the R9 Nano in Firestrike for non-SLI/Crossfire configs.  Quadro cards are not clocked as high as GeForce, but it's pretty much the same hardware, dude.

 

I believe Linus said the scores of the Nano were higher compared to his previous tests because of the water cooling.

[Marinatall_Ironside]

3DMark is down for me right now, but when it comes back compare the M4000 to the R9 Nano, I checked before posting and found reports of 16xxx for the M4000 vs 14xxx for the R9 Nano in Firestrike for non-SLI/Crossfire configs.  Quadro cards are not clocked as high as GeForce, but it's pretty much the same hardware, dude.

I know it's the same core and such (e.g. The M6000 is based on the Titan X), but some of the other ones has ECC VRAM which is slower like the RAM for the whole system because of that ECC registration to prevent errors from slipping through and causing an interruption or some stupid shit like that. However you also have to factor in the cost-to-performance for gaming, and that's where Quadros fall flat on its face compared to their gaming equivalents. Also Quadros, like AMD's FirePro cards are intended to attain maximum render quality and precision and not the maximum FPS in games.

[Marinatall_Ironside]

I believe Linus said the scores of the Nano were higher compared to his previous tests because of the water cooling.

That's correct.

[aw_]

That's correct.

 

By 800 points, which doesn't close the gap between what I saw on 3DMark, but those were the peaks, maybe average scores are closer.  Regarding price/performance, this was a THIRTY THOUSAND dollar build.  The parts lists earlier in the thread are quoting $618 per Nano, plus $123 per card for the water block, plus miscellaneous other water cooling components.  A few years ago a 4000 series Quadro would have be $1500-2000, and your price/perf argument makes sense... I'm seeing the M4000 for $800-900 now, or less if you want to gamble on a seller.  For the scale of this build, that's practically the same thing.  FUD about ECC memory and drivers is just that.

[Catsrules]

This was a cool video to see what the current generation hardware can handle.

 

In the next video I would be very insterested in compairing 1 $1500-2000 computer against 3 $500-650 computers. 

 

For example if you had a family or group of friends who lived together, would it be better for them to all go in and buy a 1500-2000 computer, or each buy there own $500-600 computer?

[LinusTech]

By 800 points, which doesn't close the gap between what I saw on 3DMark, but those were the peaks, maybe average scores are closer. Regarding price/performance, this was a THIRTY THOUSAND dollar build. The parts lists earlier in the thread are quoting $618 per Nano, plus $123 per card for the water block, plus miscellaneous other water cooling components. A few years ago a 4000 series Quadro would have be $1500-2000, and your price/perf argument makes sense... I'm seeing the M4000 for $800-900 now, or less if you want to gamble on a seller. For the scale of this build, that's practically the same thing. FUD about ECC memory and drivers is just that.

I wouldn't run 7 graphics cards next to each other on air and I don't see a block available for those cards, and even of there was, many modern water blocks are not true single slot - another reason the R9 Nano was chosen.

I think Jon has a blog post coming about some of the trials and tribulations including AMD's nonsense the last requires the host to be rebooted when a VM needs to be rebooted.

Truthfully it seemed like a minor detail in the context of this video. I don't expect anyone to build this (or anything similar) ever.

Btw thanks so much for your help with this project. Jon told me you were awesome. I have a contact at AMD who might be able to help us get the information we need to improve Fiji's behavior with VMs (assuming it's not a straight up hardware limitation)

[kameshss]

7 980 Ti's wouldn't even fit in there since most of them are dual-slot cards. Even the K|NGP|N version of those cards would be extremely expensive, both in terms of price and power consumption, let alone he would have needed to go with a CaseLabs MAGNUM case and dual power supplies to even install and power up the system, plus extra radiators to properly cool the 300-watt TDP K|NGP|N cards.

So it's possible.

 

Power Consumption? Price? This is LinusTechTips, that's how we do it  ha ha

[unijab]

Can I borrow it to heat my house this winter?

 

@LinusTech has this video broken any records for views in one day?

[byalexandr]

Can I borrow it to heat my house this winter?

 

@LinusTech has this video broken any records for views in one day?

It hit over 1,000,000 views in less than 20 hours.

[PuSHiSLaND]

this sounds like a pc i would buy for the downstairs gaming room in the LA mansion that i would buy if i won the lotto.. wow, so sexy

[monkey10120]

Any reason Unraid runs so much better then most hypervisors out there? Using VMware and Citrix at least on the enterprise side I have never been able to pull performance like Unraid is on VMS.

[unijab]

@LinusTech ... doesnt caselabs make a dual motherboard tower...

Could you of used two motherboards in a dual MB case that would of supported 5 GPUs each and had a 10 person gaming tower?

[52ObiB]

For development I've been using an NVIDIA K2200. That works great with the NVIDIA drivers in Windows and Linux VMs. No reset issues or NVIDIA driver shenanigans.

 

An AMD R9 290 has worked great too since aw_ (assuming it's the author of vfio.blogspot.com) and others got resetting working for it. I hope that the R9 Nano's get something sorted for them too.

 

I've been using ArchLinux (previously Ubuntu) as the host OS. I've not tried using LimeTech but it seems to have worked out and might have less overhead to employ for someone getting started.

 

One thing I've used is SR-IOV virtual functions off an Intel I350 NIC to make sure networking latency and overhead is low. It's probably a very slight optimization for gaming only but might make a bigger difference for NAS and router VMs.

 

I also use BTRFS CoW features. I have that on NVMe Intel 750s. The latency and bandwidth is excellent and the interrupt overhead is low (MSI-X interrupts spread across cores).

 

For anyone looking to dig into how to roll this on your own vfio.blogspot.com is probably the best place to start to get familiar with this. The mailing list is active and useful too.

 

I love the water cooling! I wish I could do that on my primary host. For me however, for work stuff, between swapping various GPUs, storage, and NICs, it's a bit impractical.

 

Very nice build Linus! Fun video to watch!

[Guest]

Any reason Unraid runs so much better then most hypervisors out there? Using VMware and Citrix at least on the enterprise side I have never been able to pull performance like Unraid is on VMS.

Because it's KVM. 

[Guest]

wow i really want to get a crazy unraid build done soon

Just build a regular Linux box and install KVM. 

[iPolymer]

I can't wait to see this being used in CES, you better make videos of it on LTT and SuperFun

[monkey10120]

Because it's KVM. 

 

But what makes KVM so much better performance wise? Because its kernal based?

[aw_]

I wouldn't run 7 graphics cards next to each other on air and I don't see a block available for those cards, and even of there was, many modern water blocks are not true single slot - another reason the R9 Nano was chosen.

I think Jon has a blog post coming about some of the trials and tribulations including AMD's nonsense the last requires the host to be rebooted when a VM needs to be rebooted.

Truthfully it seemed like a minor detail in the context of this video. I don't expect anyone to build this (or anything similar) ever.

Btw thanks so much for your help with this project. Jon told me you were awesome. I have a contact at AMD who might be able to help us get the information we need to improve Fiji's behavior with VMs (assuming it's not a straight up hardware limitation)

 

True, 7 cards would be really high density for air cooling side-by-side and now that 3dmark is back online I see that the 16k firestrike number was a fluke on the M4000, typical numbers are half of the R9 Nano.  Still, there's something to be said for a card that the vendor actually intends and supports in this configuration, at least on the right host Linux distro.

 

It's too bad we couldn't come up with a reset that works better, having a card in hand and AMD folks to consult with is pretty much a necessity for that, if indeed it is possible on the hardware.  I was glad to offer what little help I could, it really is a cool proof of concept and fun video.  Certainly nobody is going to replicate this build to the same scale, but with over a million views in the first day, even a very small percentage of viewers picking this card based on the evidence presented here is potentially a lot of unhappy users that I get to deal with later.

[mrekko]

If all the systems are all sharing the 2 Intel I210-AT Gigabit LAN Controllers on the board, wouldn't that mean just 284MBPS per system? Or did Linus slip in a 10G card somewhere i didn't see?

 

I suppose 300M would be enough for gaming, though.

[Dr_Pleeper]

I'm at a loss for words, that is truly incredible!

[DrDoxic]

Hey man i've beend dying to set up a 2 gamers 1 pc thinghy(so i can play with my gf LOL) and i encountered a lot of problems...if you can give me a few tips that'd be great

I got i7 4790k clocked at 4.6ghz, gtx 960 4gb, 2 monitors full hd, 1 hdd 7200rpm 1TB 1 ssd 250 Gb, 16 gb of RAM...My problem was the software, couldn't make windows run 2 instances with 2 mouses and 2 keyborards

[limetechjon]

I also use BTRFS CoW features. I have that on NVMe Intel 750s. The latency and bandwidth is excellent and the interrupt overhead is low (MSI-X interrupts spread across cores).

 

I'm glad to hear you saying that.  We are just about to start testing with NVMe and we were worried about using btrfs due to this spec sheet from Intel (scroll to the bottom about File System Recommendations):  

 

Curious how performance holds up over time and how you configured TRIM given that they don't want the filesystem to manage discard.  Also curious if you just threw btrfs on there on a whim and it's just been working out or if you know something we don't about the safety of using btrfs on NVMe devices.

[limetechjon]

Hey man i've beend dying to set up a 2 gamers 1 pc thinghy(so i can play with my gf LOL) and i encountered a lot of problems...if you can give me a few tips that'd be great

I got i7 4790k clocked at 4.6ghz, gtx 960 4gb, 2 monitors full hd, 1 hdd 7200rpm 1TB 1 ssd 250 Gb, 16 gb of RAM...My problem was the software, couldn't make windows run 2 instances with 2 mouses and 2 keyborards

 

Did you ever post over on our forums at lime-technology.com/forum for assistance?

[limetechjon]

True, 7 cards would be really high density for air cooling side-by-side and now that 3dmark is back online I see that the 16k firestrike number was a fluke on the M4000, typical numbers are half of the R9 Nano.  Still, there's something to be said for a card that the vendor actually intends and supports in this configuration, at least on the right host Linux distro.

 

It's too bad we couldn't come up with a reset that works better, having a card in hand and AMD folks to consult with is pretty much a necessity for that, if indeed it is possible on the hardware.  I was glad to offer what little help I could, it really is a cool proof of concept and fun video.  Certainly nobody is going to replicate this build to the same scale, but with over a million views in the first day, even a very small percentage of viewers picking this card based on the evidence presented here is potentially a lot of unhappy users that I get to deal with later.

 

There will definitely be follow-up on the AMD reset issues. I am even stating in my behind-the-scenes blog write-up that I wouldn't necessarily recommend this for anyone to recreate on their own until that issue is resolved. I think AMD has a huge opportunity to embrace virtualization for the consumer-space, but we need to get past these silly bugs. You shouldn't have to patch QEMU every time a new AMD chipset comes out. UEFI support should be ubiquitous. And on the CPU side, it'd be nice if we had an equivalent of Intel's ARK site for AMD to clearly compare processor speeds, specs, and features to filter for things like IOMMU support and core count.

[aw_]

Hey man i've beend dying to set up a 2 gamers 1 pc thinghy(so i can play with my gf LOL) and i encountered a lot of problems...if you can give me a few tips that'd be great

I got i7 4790k clocked at 4.6ghz, gtx 960 4gb, 2 monitors full hd, 1 hdd 7200rpm 1TB 1 ssd 250 Gb, 16 gb of RAM...My problem was the software, couldn't make windows run 2 instances with 2 mouses and 2 keyborards

Did you ever post over on our forums at lime-technology.com/forum for assistance?

 

Or the qemu-discuss mailing list https://lists.nongnu.org/mailman/listinfo/qemu-discuss

Or the vfio-users mailing list  https://www.redhat.com/mailman/listinfo/vfio-users

Or the help channels for your Linux distribution since the software is ubiquitous