Folding at home GPU Rig

[LazyDev]

 

 

CPU:

Ryzen 7 3700X

 

RAM: 32GB 4X CMK16GX4M2Z3600C18 @ 3200 MHz

 

Storage:

1TB NVME WD Blue

 

GPUs:

RTX 2080 Super @ x4

RTX 2070 @ x4

GTX 1070 @ x4

GTX 1070 @ x4

GTX 1070 @ x1

RTX 2060 @ x4

RTX 2060 @ x4

 

x1 slot available (Killed my 1060 in this slot, and is currently untested and unused)

 

PSUs:

Corsair HX1000i

EVGA 550GM

 

AIO Cooler:

DeepCool Gammaxx L240T

 

PCI_E0 bifurcation card:

https://riser.maxcloudon.com/en/bifurcated-risers/22-bifurcated-riser-x16-to-4x4-set.html

 

Case:
Thermaltake Level 20 GT

 

Motherboard:
MSI X570-A Pro

 

Updated, 23rd August, 2020.

[RAM555789]

Really amazed how you managed to get everything to fit into such a small case and the cable management is awesome. What are your thermals looking like when its all closed up and running?

[Guest]

[piratemonkey]

That's too much GPU. 

 

Impressed how you could cram all those cards in their.

[LazyDev]

1 minute ago, RAM555789 said:

Really amazed how you managed to get everything to fit into such a small case and the cable management is awesome. What are your thermals looking like when its all closed up and running?

Temperatures aren't too bad, considering that they''re (Was) 8 GPU's in this case. Still monitoring, tho.

[Alex Atkin UK]

8 minutes ago, LazyDev said:

Temperatures aren't too bad, considering that they''re (Was) 8 GPU's in this case. Still monitoring, tho.

What's the power consumption?  I'd be dubious that with all those GPUs it comes under 1000W.

Oh snap, I missed the second PSU.   Still curious about the load on each.

[Windows7ge]

GPU zip tied over the CPU socket 10/10

[LazyDev]

Just now, Alex Atkin UK said:

What's the power consumption?  I'd be dubious that all those GPUs come under 1000W.

Previously, I was running 4 GPU's, RTX 2080 Super, RTX 2070, GTX 1070 and the GTX 1060. The max peak power was 925 Watts total system draw. It was pushing the PSU, which is why I installed the EVGA 550GM. This PSU only powers the 4 GPU's on the front of the case. Current system draw is probably over 1.1 KW at this time.

[Alex Atkin UK]

19 minutes ago, LazyDev said:

Previously, I was running 4 GPU's, RTX 2080 Super, RTX 2070, GTX 1070 and the GTX 1060. The max peak power was 925 Watts total system draw. It was pushing the PSU, which is why I installed the EVGA 550GM. This PSU only powers the 4 GPU's on the front of the case. Current system draw is probably over 1.1 KW at this time.

I hope you live in a cold climate, I had to stop folding when summer kicked in, then when the weather cooled a bit my PSU developed coil whine so I'm waiting on a replacement which is bad as good PSUs are all completely out of stock right now.

What's the PPD for the different cards and does the slower PCIe slots make much difference?

[Alex Atkin UK]

24 minutes ago, Windows7ge said:

GPU zip tied over the CPU socket 10/10

10/10, would not re-paste CPU cooler any time soon.

[LazyDev]

6 minutes ago, Alex Atkin UK said:

What's the PPD for the different cards and does the slower PCIe slots make much difference?

RTX 2080 Super, 1.8 - 2.7M PPD

RTX 2070, 1.2-2M PPD

GTX 1070, 460 - 970K PPD

GTX 1060, 300 - 420K PPD

P106-090, 190 - 256K PPD

 

Since the release of the GPU scheduling update, the PCIe bandwidth use has dropped. I had no issues running a 1070 on a gen 3 x1 lane.

[Windows7ge]

In the future might I suggest a chassis like this.

[Alex Atkin UK]

3 hours ago, LazyDev said:

RTX 2080 Super, 1.8 - 2.7M PPD

RTX 2070, 1.2-2M PPD

GTX 1070, 460 - 970K PPD

GTX 1060, 300 - 420K PPD

P106-090, 190 - 256K PPD

 

Since the release of the GPU scheduling update, the PCIe bandwidth use has dropped. I had no issues running a 1070 on a gen 3 x1 lane.

Oh that's interesting, it never occurred to me the hardware GPU scheduling would impact folding.  None of my PCs have gotten the latest update although I usually fold on Linux anyway, PPD is generally higher than Windows.

[LazyDev]

6 minutes ago, Alex Atkin UK said:

Oh that's interesting, it never occurred to me the GPU scheduling would impact folding.  None of my PCs have gotten the latest update although I usually fold on Linux anyway, PPD is generally higher than Windows.

 

 

 

[GrockleTD]

Holy Toledo... Shut up and take my internet points

[GrockleTD]

4 hours ago, Windows7ge said:

In the future might I suggest a chassis like this.

Maybe not that specific PSU though

[Windows7ge]

7 hours ago, GrockleTD said:

Maybe not that specific PSU though

No. The hardware inside the chassis is just for demonstration purposes. Only the box itself would show up. Definitely use your own preference of PSU.

[Alex Atkin UK]

3 hours ago, Windows7ge said:

No. The hardware inside the chassis is just for demonstration purposes. Only the box itself would show up. Definitely use your own preference of PSU.

The reviews don't exactly sound good and looking at the picture I can see the flaws they mentioned.

What good is it having the intake fans blowing on the exhale ports of the GPUs?  That makes no sense.  Also sod-all airflow between the cards.

[Windows7ge]

1 hour ago, Alex Atkin UK said:

The reviews don't exactly sound good and looking at the picture I can see the flaws they mentioned.

What good is it having the intake fans blowing on the exhale ports of the GPUs?  That makes no sense.  Also sod-all airflow between the cards.

I agree the chassis design isn't ideal. You would not want to use any blower style cards in this box.

 

But I mean, OP doesn't have to use Linus's best practices of solving problems with zipties. I'd say it's a step in the right direction compared to the current configuration which as I can see all the cards mounted to the front of the chassis are in the same configuration as the 4U box so I don't think it'd make a difference to OP.

 

It'd definitely make the box more manageable.

[Alex Atkin UK]

1 minute ago, Windows7ge said:

I agree the chassis design isn't ideal. You would not want to use any blower style cards in this box.

 

But I mean, OP doesn't have to use Linus's best practices of solving problems with zipties. I'd say it's a step in the right direction compared to the current configuration which as I can see all the cards mounted to the front of the chassis are in the same configuration as the 4U box so I don't think it'd make a difference to OP.

 

It'd definitely make the box more manageable.

His setup has a much bigger gap between the GPUs.

[Windows7ge]

2 minutes ago, Alex Atkin UK said:

His setup has a much bigger gap between the GPUs.

I suppose compensating with some Delta fans isn't something he'd want to sit next to.

 

Though I'm willing to bet as it stands the box probably isn't the quietest thing.

[LazyDev]

6 minutes ago, Windows7ge said:

I suppose compensating with some Delta fans isn't something he'd want to sit next to.

 

Though I'm willing to bet as it stands the box probably isn't the quietest thing.

4 200mm fans, and 1 140mm fan. As the GPU's are spaced, plenty of airflow. System isn't too loud, and comfortable to sit next to, even whilst gaming and watching movies.

[Windows7ge]

1 minute ago, LazyDev said:

4 200mm fans, and 1 140mm fan. As the GPU's are spaced, plenty of airflow. System isn't too loud, and comfortable to sit next to, even whilst gaming and watching movies.

Probably not something I'd sleep next to but that'd be tolerable with headphones IMO.

 

That's another downside of the tighter profile of using a server form factor. Using higher RPM fans that are consequently louder.

[LazyDev]

1 minute ago, Windows7ge said:

Probably not something I'd sleep next to but that'd be tolerable with headphones IMO.

I sleep with this, running 24/7. It's no louder than gentle wind. It's not disturbing at all. No fan wines, or coil wines etc. It's comfortable. 

[Windows7ge]

3 minutes ago, LazyDev said:

I sleep with this, running 24/7. It's no louder than gentle wind. It's not disturbing at all. No fan wines, or coil wines etc. It's comfortable. 

I'm a very light sleeper. Something about the volume of a human conversation would keep me awake.

 

Good to hear it works for you though. Now what's that electric bill look like?