IT WORKS!!! - Six 8K Workstations, 1 CPU Finale

[jakkuh_t]

We finally finish and test the 6 8K editors, 1 CPU build....

 

 

[barondeau]

Finally, took long enough!

[Neamow]

I did not think they would actually finish this successfully, or that if they did it would have even close to comparable performance to separate machines. Damn.

 

However my ever present question with crazy builds like this, and this one in particular - what are they going to do with it? Since they're not actually going to use it, are they going to just disassemble it? There's no way they can recoup the cost of this build with just the videos about it.

[Nystemy]

This type of build seems a bit different to say the least.

Maybe not the most practical thing in the world, unless one were to add resource sharing between the VMs, so that resources that aren't used by one editor can be used by another and so forth. Though, at some point the question comes along if it wouldn't just be easier to use remote desktop to access a rig in a server room.

 

But regardless, interesting to see what one can do, even if it might not be the most practical thing in the world.

 

As a side note, the link to the Forum thread in the video description over at Youtube doesn't exist at current.

[Hunter259]

I still would like to know why RDS wasn't considered for this project using Windows Server 2019. It can do some pretty sweet utilization limits nowadays. Of course it requires either Tesla's or Quadro's to enable the hardware acceleration over remote desktop but it works extremely well in my system.

[Mr. Smiley]

Anyone know what chair Alex(the white chair) is using?

[microsoftenator]

I'm surprised they didn't use 1GB hugepages given that that should be supported and would reduce the overhead even further compared to 2MB pages. Transparent hugepages has issues when memory starts getting fragmented, but shouldn't really add any overhead for a VM setup unless you're utilizing memory ballooning (which is incompatible with PCI passthrough and therefore irrelevant for this setup)

 

I'm still a little confused about the use of unraid given that there doesn't appear to be much, if any, utility it provides over existing libvirt/kvm management tools, but I guess their support might be cheaper than Red Hat's? If they had gone for the Quadros, the Xen hypervisor probably would have ended up having less issues as well.

 

1 hour ago, Nystemy said:

Maybe not the most practical thing in the world, unless one were to add resource sharing between the VMs, so that resources that aren't used by one editor can be used by another and so forth.

As it's a multi-socket system, NUMA becomes an issue there, plus sharing real CPU cores between VMs generally creates pretty unpleasant latency spikes in my experience.

[Nystemy]

1 hour ago, microsoftenator said:

As it's a multi-socket system, NUMA becomes an issue there, plus sharing real CPU cores between VMs generally creates pretty unpleasant latency spikes in my experience.

Well, preferably we would limit our resource sharing to only be applicable on the same socket.

Though, if there is current software support for such is a totally different question, but on paper it should be trivial, in practice, it is likely a nightmare without end in sight....

 

One can also take latency dependency of a given task into consideration when sharing resources, but I haven't really seen this out in the wild yet. _{Not even on paper.}

[poochyena]

I had always wondered why this type of set up wasn't more common, especially in a library type setting where much power isn't needed. Now I see why. They didn't even mention the fact that if 1 computer fails, it takes down EVERYONE, not just one person, so there is a much bigger risk.

[ahuckphin]

displayport is supposed to be fantastic. but i guess linus is right, DVI is best. 

[GimmeGaming]

12 hours ago, microsoftenator said:

I'm surprised they didn't use 1GB hugepages given that that should be supported and would reduce the overhead even further compared to 2MB pages. Transparent hugepages has issues when memory starts getting fragmented, but shouldn't really add any overhead for a VM setup unless you're utilizing memory ballooning (which is incompatible with PCI passthrough and therefore irrelevant for this setup)

 

I'm still a little confused about the use of unraid given that there doesn't appear to be much, if any, utility it provides over existing libvirt/kvm management tools, but I guess their support might be cheaper than Red Hat's? If they had gone for the Quadros, the Xen hypervisor probably would have ended up having less issues as well.

 

As it's a multi-socket system, NUMA becomes an issue there, plus sharing real CPU cores between VMs generally creates pretty unpleasant latency spikes in my experience.

Did they not have a partnership with unraid or at one stage Linus was contemplating buying a stake in them. I could be wildly inaccurate but I presumed this was the reason as their would be better options.

[devuli]

18 hours ago, jakkuh_t said:

We finally finish and test the 6 8K editors, 1 CPU build....

 

 

Exactly what's the SKU of the mb? The Tyan S7100 has only 4 PCI-E 16x, and you guys put 6 gpus in there...
I have a S7105 in my lab, love it!

[Velcade]

4 hours ago, bindydad123 said:

displayport is supposed to be fantastic. but i guess linus is right, DVI is best. 

Displayport is fine as long as you buy cables that meet spec.

 

Cheap cables cause flickering. 

[alpha754293]

You need to enable SMB Direct and RDMA on both your server and this system.

 

Upside: the ACTUAL data transfers will move a lot faster.

 

Downside: Windows Task Manager won't be able to see the data transfer rates BECAUSE it's RDMA/SMB Direct.

 

You guys should also really look into either picking up a relatively inexpensive 18 port 100 GbE Mellanox switch or if that's too rich for your blood, at least run an AOC direct attach cable so that you can run 100 Gbps from the server to this directly.

 

Also, DON'T use Intel Optane PCIe SSDs for this.

 

You WILL wear through the write endurance of the drives, I can guarantee that; at which point, the drives will go into a read-only state and will only write at 2 MB/s.

 

I know because I just received my RMA'd Intel 750 series PCIe AIC SSD back because I wore through the write endurance limit on the drive/card.

 

Use Micron 9300 Max U.2 NVMe drives instead.

[Fristad]

How come you're still using Unraid? It's such an awful solution in comparison to thin clients and Citrix Xenserver or VMWare ESXi with GPU-passthrough. 

[williamcll]

Tyan didn't hesitate to boast

[devuli]

6 hours ago, williamcll said:

Tyan didn't hesitate to boast

Thats not a standard S7100 like we see on Tyan website... I will contact Tyan!

[jakkuh_t]

On 7/29/2019 at 6:22 AM, devuli said:

Exactly what's the SKU of the mb? The Tyan S7100 has only 4 PCI-E 16x, and you guys put 6 gpus in there...
I have a S7105 in my lab, love it!

https://www.tyan.com/Motherboards_S7100-EX_S7100AGM2NR-EX

[devuli]

On 8/1/2019 at 8:34 PM, jakkuh_t said:

https://www.tyan.com/Motherboards_S7100-EX_S7100AGM2NR-EX

yes! but it has 4 pcie 16x, how you put gpu in a 8x slot?

[Results45]

Question: It works, but is it actually now the primary workstation everyone at LTT edits videos today?