[Quad RTX2080Ti + Threadripper 2] Project Windtunnel(s)

[For Science!]

It's that time again to upgrade our office workstations! For Science!

 

We have been having a great experience using these quad-GPU machines + Threadripper for our scientific calculations (structural biology, for those curious). It is of course very rare to see quad-GPU machines for gaming nowadays so I thought it is good for some documentation for those crazy enough to stumble across these kind of ideas. We have to use NVIDIA cards since our programs are only CUDA compatible, and we can't use Tensor cores, RT cores, or double precision, and so the GeForce GTX series are still the most ideal. We also have a relatively steep memory capacity requirement and so need at least 11 GB VRAM - this favors 1080 Tis over 2080's.

 

 

Since the release of the second generation Threadripper, we wanted to roll out  these quad-GPU machines further in our group but also try to make some improvements. Mainly in the department of GPU thermals and as a byproduct, noise (custom watercooling all the machines would be ideal, but is not an option for now). Before rolling out on a massive purchase, we first assembled one machine as a proof of principle using the specs below. Notably we use a riser cable to bring the 2nd GPU out into the vertical slot of the View 37, this means that only 1 graphics card is choked as opposed to 3 cards being choked in a standard layout. Also, we opted for a QVL certified kit of memory since getting 128 GB of memory to work on the first generation was a bit of a nightmare that I didn't want to re-live.

 

Going to the View 37 Case also reduced the footprint of our workstation from the cube that is the 740D while also allowing for the vertical GPU mount that helps in thermals (and noise) in this very peculiar scenario. 

 

Assembled Prototype:

CPU: AMD Ryzen Threadripper 2990WX

Motherboard: ASRock Fatal1ty X399 Professional Gaming

Memory: G. SKILL Flare X (for AMD) F4-2400C15Q2-128GFX (128 GB, 2400 MHz, CL15)

CPU Cooling: Noctua NH-U14S TR4 SP3 in Push-Pull

GPU: Gigabyte GeForce GTX 1080 Ti Turbo 11G in Quad Layout

PSU: Corsair AX1600i 

Case: Thermaltake View 37 Riiing Edition

Fans: Fractal Design Venturi HP-14 PWM 140 mm

Extra: Thermaltake TT Premium PCI-E 3.0 Extender 300 mm

Storage: 500 GB + 1 TB SATA SSD

 

I was largely pleased with this build and decided to proceed with the other builds with some minor adjustments. Getting hold of 1080Tis are getting more difficult for everybody, and so we'll see what we end up with. CPU cooler wise, I could obtain the Dark Rock Pro TR4 for a similar price, and so went with them since the performance is similar while subjectively is more appealing to the eye.

 

Finalized Specs (build in progress....):

CPU: AMD Ryzen Threadripper 2950X (mostly) or 2990WX

Motherboard: ASRock Fatal1ty X399 Professional Gaming

Memory: G. SKILL Flare X (for AMD) F4-2400C15Q2-128GFX (128 GB, 2400 MHz, CL15)

CPU Cooling: Dark Rock Pro TR4 in Push-Push-Pull

GPU: ASUS GeForce GTX 1080 Ti Turbo 11G in Quad Layout

PSU: Corsair AX1600i 

Case: Thermaltake View 37 Riiing Edition

Fans: Fractal Design Venturi HP-14 PWM 140 mm (some machines receive stock Riiing fans)

Extra: Thermaltake TT Premium PCI-E 3.0 Extender 300 mm

Storage: 500 GB + 1 TB SATA SSD

 

 

More to come....

[tintexas]

What's the final verdict on this build?  

 

Is there enough airflow for you?  Can you, uh, turn off all the blinking lights? Do you wish there was a second vertical mount?

[For Science!]

20 hours ago, tintexas said:

What's the final verdict on this build?  

 

Is there enough airflow for you?  Can you, uh, turn off all the blinking lights? Do you wish there was a second vertical mount?

I should have some updates soon, we had some definite hiccups with regards to obtaining GPUs.

There are BIOS options to turn off the LEDs (although there aren't too many to begin with). I do indeed wish there was a second vertical mount, currently the GPU on the vertical mount is the coolest and the card that is back-to-back with the bottom slot card is still getting really hot (and loud). But It's definitely an improvement over a normal quad layout.

 

[For Science!]

Just to give this thread some closure, since I really didn't have much time lately.

In the end due to various availability issues, our hands were forced into purchasing a crap tonne of RTX 2080 Tis instead of the 1080 Tis. As before, since these are quad-GPU workstations blower style cards were desired. I initially thought "All blower cards are the same" and so procured some Gigabyte Turbo as well as Asus Turbo cards for faster purchasing. This was quite a big mistake, so read on but the TL:DR is: Gigabyte Turbo cards are rubbish compared to Asus Turbo.

 

Crap tonne of card boxes

Crap tonne of cards, unpackaged

 

Example installation of one machine with four cards

 

Not copy paste, I swear

 

Now, after assembling and sealing the workstations, we came to notice that the Gigabyte Turbo cards were consistently louder AND hotter, than the Asus Turbo cards. This was genuinely a heat issue since the Asus cards were also able to maintain higher clocks. Since we had 12 Gigabyte cards and 8 Asus cards, we also were in a unique position to have a sample size big enough to rule out individual differences - The Gigabyte cards were consistently worst (Hotter, Slower, and Louder).

 

In the end we had to swap the Gigabyte Cards for Asus ones while the loud cards will get relocated into a server room context where noise isn't an issue. In short my personal lesson was that not all blower cards are built equal. Either the Asus cooler was somehow genuinely better than the Gigabyte one, or that the voltage supplied to the Gigabyte is just crazy, but I was not going to tweak these individually since these workstations need to be pretty stable.

 

Anyhow, that was the end of that.

[_Hustler_One_]

THAT ARE SOME SCARY BLACK OUT!

 

I love black. And aircooled.

[For Science!]

As a side note on temperatures, the GPU temps were fairly okay and went in the order of: Side Mounted GPU < Bottom GPU < Top GPU <<<< Sandwiched GPU. As expected the GPU that is choked for air was the hottest. This made me try to look for a motherboard that had a x16 slot in between position 2 and 3, such as some of the MSI and Gigabyte boards. However these board's middle lane are always connected to the chipset instead of the CPU (since the intended use is for a network card or SSD mount) and so I judged it would not be suited for our needs.

[mattpril]

Hi great builds! Did you find any thermal throttling at full capacity when running for long (10-24+) hours? I'd appreciate any insight! Thanks!

[For Science!]

6 minutes ago, mattpril said:

Hi great builds! Did you find any thermal throttling at full capacity when running for long (10-24+) hours? I'd appreciate any insight! Thanks!

Thermal throttling as in "not boosting" yes, the choked GPU usually runs at the base clock. However it doesn't go below base cloc kperformance ever. CPU is completely fine.

[mattpril]

BRILLIANT! Thanks for the quick response. Im curious if you could mount the choked GPU vertically onto the rear fan bracket (screw holes) infront of the CPU cooler... not sure if theres proper width clearance for another GPU, or whatll happen with the extra heat onto the back of the card.. May just have to go with a low profile AIO... 

[For Science!]

On 10/24/2019 at 8:39 PM, mattpril said:

BRILLIANT! Thanks for the quick response. Im curious if you could mount the choked GPU vertically onto the rear fan bracket (screw holes) infront of the CPU cooler... not sure if theres proper width clearance for another GPU, or whatll happen with the extra heat onto the back of the card.. May just have to go with a low profile AIO... 

Possible, but I did not want to do any mods since we were building 8 workstations at the time. If I had the budget, I would consider getting a hybrid card and put that as GPU position 1, which then becomes independent of airflow which can be mounted on the rear.

[redbread123]

I could sell every organ in my body and still not be able to afford that.

[Mbowen]

On 10/29/2019 at 8:05 PM, redbread123 said:

I could sell every organ in my body and still not be able to afford that.

Isn't a kidney or lobe of liver like $80-$100k? I know this is an exaggeration but still... 

I'd sell your organs for that setup :]

[redbread123]

1 hour ago, Mbowen said:

Isn't a kidney or lobe of liver like $80-$100k? I know this is an exaggeration but still... 

I'd sell your organs for that setup :]

Honestly, I'd be willing to sell a kidney, help out anyone with kidney failure and get 10k 

[zogthegreat]

On 6/16/2019 at 7:37 PM, For Science! said:

Not copy paste, I swear

 

 

(sigh) I hate you!

 

Can I have one please? 

 

BTW, it's simply amazing how much computational power you can have siting on your desktop these days. Good luck with your research.

[Drama Lama]

I am just speechless 

[Letgomyleghoe]

On 11/19/2019 at 8:38 AM, Mbowen said:

Isn't a kidney or lobe of liver like $80-$100k? I know this is an exaggeration but still... 

I'd sell your organs for that setup :]

Hmm ideas...

 

but awesome build, are they In Their finished state how they are now? Or are you planning to water cool them all?

[For Science!]

1 minute ago, Letgomyleghoe said:

Hmm ideas...

 

but awesome build, are they In Their finished state how they are now? Or are you planning to water cool them all?

I've actually moved away from this particular work place, but I have been told that they are happily living their lives out doing science. I think one riser cable had an issue, and one motherboard died out prematurely and stopped recognizing all memory sticks. Watercooling all of them out be insanely out of budget and out of the technical expertise now that I am no there anymore

 

So yes, they will probably be run down into the ground and then probably one day be re-purposed for teaching purposes or something like that.

[Letgomyleghoe]

15 minutes ago, For Science! said:

So yes, they will probably be run down into the ground and then probably one day be re-purposed for teaching purposes or something like that.

R.I.P 15,000 pcs, I’m hoping someone there will at least take them apart and perform maintenance.

[For Science!]

2 minutes ago, Letgomyleghoe said:

R.I.P 15,000 pcs, I’m hoping someone there will at least take them apart and perform maintenance.

The left over guys were adept enough to replace motherboards and riser cables so I'm sure It'll be fine. Also why I kept them in eye-line together with a very windowed panel since "if its out of sight, its out of mind" and so if its within eyesight, the chance that they receive occasional TLC is higher.

[Letgomyleghoe]

2 hours ago, For Science! said:

Also why I kept them in eye-line together with a very windowed panel since "if its out of sight, its out of mind" and so if its within eyesight, the chance that they receive occasional TLC is higher.

Honestly that’s the only reason why I like my tempered glass so much, I looking over and seeing a dusty computer drives me up a wall.