Anyone tried a dual Epyc workstation?

[Grabhanem]

A family member of mine is a theoretical physicist whose computer workload consists of very heavily multithreaded simulations. He's been looking for a workstation upgrade from his current dual E5-2640 v4 Broadwell system. Looking around the available options at his target budget of around $5K, the best option seems to be something like a custom dual Epyc 7352 system.

 

Does anyone have experience with this configuration? I know that the H11DSi board currently available has to be rev 2.0 to support 7002 series CPUs. Are there any other potential issues to be aware of?

128GB ECC memory (16*8GB)	$900
Assorted other stuff (GPU, drives)	$200
Case	$100
Decent 1000W power supply	$200
Dual Epyc 7352 (24C each)	$2,900
Dual SP3 motherboard	$600
Noctua heatsinks	$160

[Fasauceome]

I have heard from the high end overclockers and server enthusiasts like Der8auer that gigabyte makes the best dual epyc systems.

[GoldenLag]

32 minutes ago, Grabhanem said:

Does anyone have experience with this configuration? I know that the H11DSi board currently available has to be rev 2.0 to support 7002 series CPUs. Are there any other potential issues to be aware of?

just be aware that some cases simply wont fit most motherboards. 

 

other than that it seems fine. also a lot of motherboards have onboards graphics. 

[Grabhanem]

3 minutes ago, Fasauceome said:

I have heard from the high end overclockers and server enthusiasts like Der8auer that gigabyte makes the best dual epyc systems.

Do they offer conventional desktop form factors? He's looking for a desktop workstation, not a rack unit.

1 minute ago, GoldenLag said:

just be aware that some cases simply wont fit most motherboards. 

 

other than that it seems fine. also a lot of motherboards have onboards graphics. 

I was looking at something like the Phanteks Enthoo Pro as it claims to support SSI-EEB.

 

Using the onboard graphics is a possibilty- realistically though, the GPU isn't really a major part of the budget so I'm not too worried.

[mariushm]

I would suggest waiting for the Threadripper 3rd generation to launch.

 

You have the 2000$  Threadripper 3970x 32 core / 64 threads running at 3.7 Ghz base, 4.5 Ghz boost clock and 144 MB of cache memory : https://www.amd.com/en/products/cpu/amd-ryzen-threadripper-3970x

 

I suspect the performance will be pretty close or above two EPYCs, considering the EPYC have 2.4 Ghz base clock, and you'd also be dealing with shuffling data between processors, moving between dies, numa nodes etc.

 

You'd be able to use any plain EATX case and you should be able to use regular DDR4 ECC memory - you'll want 4 memory sticks, or 8, because these are quad channel processors.

 

Also, you have pci-e 4.0, and you have pci-e 4.0 x8 connection to the chipset, so it's less chocking at high transfer speeds.

 

These new EPYC need TRX40 motherboards, the existing TR4 boards won't be compatible.

 

The boards will probably be in the 400-600$ range.

 

The TR4 water AIO kits will probably work, so that's around 100-200$ I guess on water cooling the CPU - it's a 280w TDP cpu, so not easy to cool, but not difficult for decent water coolers.

 

Memory ... you can go up to 3200 Mhz, but if you want ECC, you'd probably stick to 2933 Mhz at most.

These will probably be supported - 32 GB 2666 ECC stick for 149$...169$ -  so you could get 4 x 32 and have 4 extra slots as reserve : https://pcpartpicker.com/product/sdZFf7/crucial-32-gb-1-x-32-gb-ddr4-2666-memory-ct32g4lfd4266

Wait until the boards actually show up, and they confirm it does 32 GB sticks though.

 

 

 

[Grabhanem]

10 minutes ago, mariushm said:

I would suggest waiting for the Threadripper 3rd generation to launch.

 

You have the 2000$  Threadripper 3970x 32 core / 64 threads running at 3.7 Ghz base, 4.5 Ghz boost clock and 144 MB of cache memory : https://www.amd.com/en/products/cpu/amd-ryzen-threadripper-3970x

 

I suspect the performance will be pretty close or above two EPYCs, considering the EPYC have 2.4 Ghz base clock, and you'd also be dealing with shuffling data between processors, moving between dies, numa nodes etc.

 

You'd be able to use any plain EATX case and you should be able to use regular DDR4 ECC memory - you'll want 4 memory sticks, or 4, because these are quad channel processors.

 

Also, you have pci-e 4.0, and you have pci-e 4.0 x8 connection to the chipset, so it's less chocking at high transfer speeds.

 

These new EPYC need TRX40 motherboards, the existing TR4 boards won't be compatible.

 

The boards will probably be in the 400-600$ range.

 

The TR4 water AIO kits will probably work, so that's around 100-200$ I guess on water cooling the CPU - it's a 280w TDP cpu, so not easy to cool, but not difficult for decent water coolers.

 

Memory ... you can go up to 3200 Mhz, but if you want ECC, you'd probably stick to 2933 Mhz at most.

These will probably be supported - 32 GB 2666 ECC stick for 149$...169$ -  so you could get 4 x 32 and have 4 extra slots as reserve : https://pcpartpicker.com/product/sdZFf7/crucial-32-gb-1-x-32-gb-ddr4-2666-memory-ct32g4lfd4266

Wait until the boards actually show up, and they confirm it does 32 GB sticks though.

 

 

 

This is definitely something I'll keep in mind, but for right now-

The workload is very multicore friendly, so performance is basically clock * cores. When calculated at both base & boost clock, the performance of 32 TR cores vs. 48 Epyc cores is pretty similar. The Epyc platform also has plenty of upgrade headroom for the future or if prices go down, and likely better ECC monitoring, etc.

 

NUMA etc. isn't really a concern- these simulations have been run on dual Xeon systems for years, so they're well equipped for that.

 

This will likely get built this spring, so a lot will depend on what the market looks like then.

[XenonXTZ]

On 11/19/2019 at 11:55 AM, Grabhanem said:

A family member of mine is a theoretical physicist whose computer workload consists of very heavily multithreaded simulations. He's been looking for a workstation upgrade from his current dual E5-2640 v4 Broadwell system. Looking around the available options at his target budget of around $5K, the best option seems to be something like a custom dual Epyc 7352 system.

 

Does anyone have experience with this configuration? I know that the H11DSi board currently available has to be rev 2.0 to support 7002 series CPUs. Are there any other potential issues to be aware of?

128GB ECC memory (16*8GB)	$900
Assorted other stuff (GPU, drives)	$200
Case	$100
Decent 1000W power supply	$200
Dual Epyc 7352 (24C each)	$2,900
Dual SP3 motherboard	$600
Noctua heatsinks	$160

Hi. Are you still looking for a similar setup?

I have the following for less than your projected price:

2x EPYC 7551

512GB 2666MHz RAM

H11DSI-NT

6x 1.92TB SSDs in RAID5

LSi 9361-8i RAID Controller with CacheVault and BBU

1x 4TB NVMe as boot

1KW PSU 80+ Titanium

Noctua NH-U12S for SP3

 

Let me know if interested.

Thank you.