3U ThreadRipper 3970x Compute Node

[Thavion Hawk]

Country: United States 

Games, programs or workloads that it will be used for: Linux Computer Node?

Other details: I don't know what the customer wants these for, but I'm building them!

 

As ever my job has steered me into building something amazing. Something not quite Epic, but truly Ripping... That's so bad I'm sorry.

We have a customer that has in the past asked for some really beefy rigs as detailed on the forums last June.

Mid March of this year they came back asking for more. Not more of the same, but simply More. By that I mean they want systems with more Cores/Threads, RAM, Storage and all of it in a 2/3U Rack mount...

Now I told them point blank that I can't go past 128GB's of RAM on anything short of HEDT and they said Yes. After a few quotes we came down to the following specs.

 

CPU: R9-3970X (32c/64t)

Cooler: Dynamix A26 (Small and fits.)

RAM: 8x32GB Crucial Ballistix 3,200Mhz CL16 (They wanted 512GB but didn't want to pay for ThreadRipper Pro)

Mobo: ASRock TRX40 Creator (Does what they want +10Gb NIC for inter node connection)

Storage-

 SSD: 1TB Samsung 970 EVO (Because they don't need a 980PRO)

 HDDs: 3 x 8TB Seagate Ironwolfs in RAID 5 (More Storage, More Better)

GPU: GT710 (No GPU Compute in these rigs.)

PSU: BeQuiet PurePower 11 500w (Enough power, 80+ Gold and Modular)

Case: iStar D-Storm 300-PFS (3U, Full ATX, Rack mount) + Extra Noctua fan for airflow.

 

The only thing I'm woried about is the small CPU Cooler, but it's rated for the job given enough airflow.

Here is the Parts Picture less the case, I'll upload the build when It's all done.

 

 

[emosun]

is this a build log or did you have a question?

[Thavion Hawk]

1 minute ago, emosun said:

is this a build log or did you have a question?

Build log. I've got the parts and figured like I've done before I'd post here as I go.

[emosun]

3 minutes ago, Thavion Hawk said:

Build log. I've got the parts and figured like I've done before I'd post here as I go.

oh ok , there is a build log section if that might interest you?

[Thavion Hawk]

1 minute ago, emosun said:

oh ok , there is a build log section if that might interest you?

You say that as I notice that I'd posted the last build for this customer in the build log section. Is there a function to move this post or would it be simpler for me to simple repost there and close this one?

 

[WhitetailAni]

11 minutes ago, Thavion Hawk said:

You say that as I notice that I'd posted the last build for this customer in the build log section. Is there a function to move this post or would it be simpler for me to simple repost there and close this one?

 

Just wait a bit. A moderator will come along and move it (I reported the post with "user asked thread be moved to build logs").

[Thavion Hawk]

1 minute ago, FakeKGB said:

Just wait a bit. A moderator will come along and move it (I reported the post with "user asked thread be moved to build logs").

Thanks. My bad.

[Crunchy Dragon]

-Moved to Build Logs-

[Thavion Hawk]

15 minutes ago, Crunchy Dragon said:

-Moved to Build Logs-

Thank You.

[Thavion Hawk]

3 hours ago, Thavion Hawk said:

Country: United States 

Games, programs or workloads that it will be used for: Linux Computer Node?

Other details: I don't know what the customer wants these for, but I'm building them!

 

As ever my job has steered me into building something amazing. Something not quite Epic, but truly Ripping... That's so bad I'm sorry.

We have a customer that has in the past asked for some really beefy rigs as detailed on the forums last June.

Mid March of this year they came back asking for more. Not more of the same, but simply More. By that I mean they want systems with more Cores/Threads, RAM, Storage and all of it in a 2/3U Rack mount...

Now I told them point blank that I can't go past 128GB's of RAM on anything short of HEDT and they said Yes. After a few quotes we came down to the following specs.

 

CPU: R9-3970X (32c/64t)

Cooler: Dynamix A26 (Small and fits.)

RAM: 8x32GB Crucial Ballistix 3,200Mhz CL16 (They wanted 512GB but didn't want to pay for ThreadRipper Pro)

Mobo: ASRock TRX40 Creator (Does what they want +10Gb NIC for inter node connection)

Storage-

 SSD: 1TB Samsung 970 EVO (Because they don't need a 980PRO)

 HDDs: 3 x 8TB Seagate Ironwolfs in RAID 5 (More Storage, More Better)

GPU: GT710 (No GPU Compute in these rigs.)

PSU: BeQuiet PurePower 11 500w (Enough power, 80+ Gold and Modular)

Case: iStar D-Storm 300-PFS (3U, Full ATX, Rack mount) + Extra Noctua fan for airflow.

 

The only thing I'm woried about is the small CPU Cooler, but it's rated for the job given enough airflow.

Here is the Parts Picture less the case, I'll upload the build when It's all done.

 

 

 

Starting with the good stuff, here's a shot of the CPU, RAM and SSD installed with the small CPU Cooler installed. Really if I didn't know they had no interest in pushing the hardware past strictly Stock, I'd have pushed to fit something a bit larger. Then again the number of TR4/TRX40 coolers that fit a 2/3U Server are limited. Low Profile cooling for 32+ Cores anyone?

 

 

As for the Case and the build as a hole? Well the Case was picked because it fits a full ATX Motherboard, Full Hight PCIe Cards and an ATX PSU. The first problem I found was the stalk layout of the case mounts the PSU in the Front Right of the Case and when installed that puts the back of the PSU 5mm or so from the SATA headers... Not only does that block the SATA headers that are needed for the 3x8TB HDDs, it also blocks the Modular Headers on the PSU itself.

 

 

Thankfully the Front Mounts both Left and Right are Swappable with a few screws letting me move the PSU to the other side of the case. Now that still leaves both of the PCIe Power Headers on the PSU blocked by the Motherboard's Primary EPS12v Header, but there is no plan that I know of to install Computer grade GPU's into this system.

 

There is also no plan to install External 3.5/5 so flipping the front bays around won't matter.

 

As for stress testing and Stability? I'm running a full cycle of MemTest86. With 256GB's of RAM that's just a good idea.

 

[RollinLower]

ooh that's an awesome build! i'm pretty curious what your client is gonna use this box for.

 

i noticed in one of our datacenters a client is using boxes like these (2/3U + consumer grade hardware) because frankly he just didn't want to pay the premium for EPYC.

it's quite amusing to see RGB lights seeping from a couple servers in a otherwise bland datacenter.

[WhitetailAni]

Please turn the two tiny fans 90 degrees towards the PCIe slots...

[Thavion Hawk]

Yeah the fan is now pulling air away from the HDD's and I've disabled the VRM Fan because it's pointless given the airflow drawn by the CPU Cooler. Plus the fact that that fan is meant to pull air towards the HDD Cage as it is orientated which would cause a pressure conflict. I also mounted a 90mm Intake Fan to pull Air into the case so that it is not relying on the Negative Pressure of the  Rear 2x60 and 1x90mm fans. I honestly failed to fully vet the compatibility of this Motherboard and Case and have to work around that or else order 5 new cases.