Jump to content
Search In
  • More options...
Find results that contain...
Find results in...
For Science!

[Quad RTX2080Ti + Threadripper 2] Project Windtunnel(s)

Recommended Posts

Posted · Original PosterOP

It's that time again to upgrade our office workstations! For Science!

 

We have been having a great experience using these quad-GPU machines + Threadripper for our scientific calculations (structural biology, for those curious). It is of course very rare to see quad-GPU machines for gaming nowadays so I thought it is good for some documentation for those crazy enough to stumble across these kind of ideas. We have to use NVIDIA cards since our programs are only CUDA compatible, and we can't use Tensor cores, RT cores, or double precision, and so the GeForce GTX series are still the most ideal. We also have a relatively steep memory capacity requirement and so need at least 11 GB VRAM - this favors 1080 Tis over 2080's.

 

20170928_153408.jpg.2878f311d6fc0b2824b2f13e4ecefc7b.jpg20171031_103605.jpg.fe805eb0d1177b00c0de1d268844d9a4.jpg

 

Since the release of the second generation Threadripper, we wanted to roll out  these quad-GPU machines further in our group but also try to make some improvements. Mainly in the department of GPU thermals and as a byproduct, noise (custom watercooling all the machines would be ideal, but is not an option for now). Before rolling out on a massive purchase, we first assembled one machine as a proof of principle using the specs below. Notably we use a riser cable to bring the 2nd GPU out into the vertical slot of the View 37, this means that only 1 graphics card is choked as opposed to 3 cards being choked in a standard layout. Also, we opted for a QVL certified kit of memory since getting 128 GB of memory to work on the first generation was a bit of a nightmare that I didn't want to re-live.

 

Going to the View 37 Case also reduced the footprint of our workstation from the cube that is the 740D while also allowing for the vertical GPU mount that helps in thermals (and noise) in this very peculiar scenario. 

 

Assembled Prototype:

CPU: AMD Ryzen Threadripper 2990WX

Motherboard: ASRock Fatal1ty X399 Professional Gaming

Memory: G. SKILL Flare X (for AMD) F4-2400C15Q2-128GFX (128 GB, 2400 MHz, CL15)

CPU Cooling: Noctua NH-U14S TR4 SP3 in Push-Pull

GPU: Gigabyte GeForce GTX 1080 Ti Turbo 11G in Quad Layout

PSU: Corsair AX1600i 

Case: Thermaltake View 37 Riiing Edition

Fans: Fractal Design Venturi HP-14 PWM 140 mm

Extra: Thermaltake TT Premium PCI-E 3.0 Extender 300 mm

Storage: 500 GB + 1 TB SATA SSD

IMG_6938.thumb.png.458a21f89b1bcd3e69d4167422e8a8ba.pngIMG_6941.thumb.png.b8b91bb4416eadf9d7b9dab833379c54.png

 

I was largely pleased with this build and decided to proceed with the other builds with some minor adjustments. Getting hold of 1080Tis are getting more difficult for everybody, and so we'll see what we end up with. CPU cooler wise, I could obtain the Dark Rock Pro TR4 for a similar price, and so went with them since the performance is similar while subjectively is more appealing to the eye.

 

Finalized Specs (build in progress....):

CPU: AMD Ryzen Threadripper 2950X (mostly) or 2990WX

Motherboard: ASRock Fatal1ty X399 Professional Gaming

Memory: G. SKILL Flare X (for AMD) F4-2400C15Q2-128GFX (128 GB, 2400 MHz, CL15)

CPU Cooling: Dark Rock Pro TR4 in Push-Push-Pull

GPU: ASUS GeForce GTX 1080 Ti Turbo 11G in Quad Layout

PSU: Corsair AX1600i 

Case: Thermaltake View 37 Riiing Edition

Fans: Fractal Design Venturi HP-14 PWM 140 mm (some machines receive stock Riiing fans)

Extra: Thermaltake TT Premium PCI-E 3.0 Extender 300 mm

Storage: 500 GB + 1 TB SATA SSD

 

IMG_6962.thumb.png.a7e58500334b7c0345404846531d3281.png

 

More to come....

Link to post
Share on other sites

What's the final verdict on this build?  

 

Is there enough airflow for you?  Can you, uh, turn off all the blinking lights? Do you wish there was a second vertical mount?

Link to post
Share on other sites
Posted · Original PosterOP
20 hours ago, tintexas said:

What's the final verdict on this build?  

 

Is there enough airflow for you?  Can you, uh, turn off all the blinking lights? Do you wish there was a second vertical mount?

I should have some updates soon, we had some definite hiccups with regards to obtaining GPUs.

There are BIOS options to turn off the LEDs (although there aren't too many to begin with). I do indeed wish there was a second vertical mount, currently the GPU on the vertical mount is the coolest and the card that is back-to-back with the bottom slot card is still getting really hot (and loud). But It's definitely an improvement over a normal quad layout.

 

Link to post
Share on other sites
Posted · Original PosterOP

Just to give this thread some closure, since I really didn't have much time lately.

In the end due to various availability issues, our hands were forced into purchasing a crap tonne of RTX 2080 Tis instead of the 1080 Tis. As before, since these are quad-GPU workstations blower style cards were desired. I initially thought "All blower cards are the same" and so procured some Gigabyte Turbo as well as Asus Turbo cards for faster purchasing. This was quite a big mistake, so read on but the TL:DR is: Gigabyte Turbo cards are rubbish compared to Asus Turbo.

 

Crap tonne of card boxes

IMG_2334.thumb.jpg.8bb3676e71f864c60ad022ddbed03886.jpg

Crap tonne of cards, unpackaged

IMG_2335.thumb.jpg.cc313f0a3c482e4d13735d328738ae98.jpg

 

Example installation of one machine with four cards

IMG_2339.thumb.jpg.0a5660ebdea4f46addb64103d0c332bd.jpg

 

Not copy paste, I swear

IMG_2342.thumb.jpg.b6196647bd1e1f35d46006f594d61b47.jpg

 

Now, after assembling and sealing the workstations, we came to notice that the Gigabyte Turbo cards were consistently louder AND hotter, than the Asus Turbo cards. This was genuinely a heat issue since the Asus cards were also able to maintain higher clocks. Since we had 12 Gigabyte cards and 8 Asus cards, we also were in a unique position to have a sample size big enough to rule out individual differences - The Gigabyte cards were consistently worst (Hotter, Slower, and Louder).

 

In the end we had to swap the Gigabyte Cards for Asus ones while the loud cards will get relocated into a server room context where noise isn't an issue. In short my personal lesson was that not all blower cards are built equal. Either the Asus cooler was somehow genuinely better than the Gigabyte one, or that the voltage supplied to the Gigabyte is just crazy, but I was not going to tweak these individually since these workstations need to be pretty stable.

 

Anyhow, that was the end of that.

Link to post
Share on other sites

THAT ARE SOME SCARY BLACK OUT!

 

I love black. And aircooled.


My system specs:

Spoiler

CPU: Intel Core i7-8700K, 5GHz Delidded LM || CPU Cooler: Noctua NH-C14S w/ NF-A15 & NF-A14 Chromax fans in push-pull cofiguration || Motherboard: MSI Z370i Gaming Pro Carbon AC || RAM: Corsair Vengeance LPX DDR4 2x8Gb 2666 || GPU: EVGA GTX 1060 6Gb FTW2+ DT || Storage: Samsung 860 Evo M.2 SATA SSD 250Gb, 2x 2.5" HDDs 1Tb & 500Gb || ODD: 9mm Slim DVD RW || PSU: Corsair SF600 80+ Platinum || Case: Cougar QBX + 1x Noctua NF-R8 front intake + 2x Noctua NF-F12 iPPC top exhaust + Cougar stock 92mm DC fan rear exhaust || Monitor: ASUS VG248QE || Keyboard: Ducky One 2 Mini Cherry MX Red || Mouse: Logitech G703 || Audio: Corsair HS70 Wireless || Other: XBox One S Controler

My build logs:

 

Link to post
Share on other sites
Posted · Original PosterOP

As a side note on temperatures, the GPU temps were fairly okay and went in the order of: Side Mounted GPU < Bottom GPU < Top GPU <<<< Sandwiched GPU. As expected the GPU that is choked for air was the hottest. This made me try to look for a motherboard that had a x16 slot in between position 2 and 3, such as some of the MSI and Gigabyte boards. However these board's middle lane are always connected to the chipset instead of the CPU (since the intended use is for a network card or SSD mount) and so I judged it would not be suited for our needs.

Link to post
Share on other sites

Hi great builds! Did you find any thermal throttling at full capacity when running for long (10-24+) hours? I'd appreciate any insight! Thanks!

Link to post
Share on other sites
Posted · Original PosterOP
6 minutes ago, mattpril said:

Hi great builds! Did you find any thermal throttling at full capacity when running for long (10-24+) hours? I'd appreciate any insight! Thanks!

Thermal throttling as in "not boosting" yes, the choked GPU usually runs at the base clock. However it doesn't go below base cloc kperformance ever. CPU is completely fine.

Link to post
Share on other sites

BRILLIANT! Thanks for the quick response. Im curious if you could mount the choked GPU vertically onto the rear fan bracket (screw holes) infront of the CPU cooler... not sure if theres proper width clearance for another GPU, or whatll happen with the extra heat onto the back of the card.. May just have to go with a low profile AIO... 

Link to post
Share on other sites
Posted · Original PosterOP
On 10/24/2019 at 8:39 PM, mattpril said:

BRILLIANT! Thanks for the quick response. Im curious if you could mount the choked GPU vertically onto the rear fan bracket (screw holes) infront of the CPU cooler... not sure if theres proper width clearance for another GPU, or whatll happen with the extra heat onto the back of the card.. May just have to go with a low profile AIO... 

Possible, but I did not want to do any mods since we were building 8 workstations at the time. If I had the budget, I would consider getting a hybrid card and put that as GPU position 1, which then becomes independent of airflow which can be mounted on the rear.

Link to post
Share on other sites
On 10/29/2019 at 8:05 PM, redbread123 said:

I could sell every organ in my body and still not be able to afford that.

Isn't a kidney or lobe of liver like $80-$100k? I know this is an exaggeration but still... 

I'd sell your organs for that setup :]

Link to post
Share on other sites
1 hour ago, Mbowen said:

Isn't a kidney or lobe of liver like $80-$100k? I know this is an exaggeration but still... 

I'd sell your organs for that setup :]

Honestly, I'd be willing to sell a kidney, help out anyone with kidney failure and get 10k 

Link to post
Share on other sites
On 6/16/2019 at 7:37 PM, For Science! said:

Not copy paste, I swear

IMG_2342.thumb.jpg.b6196647bd1e1f35d46006f594d61b47.jpg

 

 

(sigh) I hate you!

 

Can I have one please? 😁

 

BTW, it's simply amazing how much computational power you can have siting on your desktop these days. Good luck with your research.

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now


×