Dual AMD Epyc based Engineering Workstation

[AerospaceDoctor]

First I note the title change based on great input, hoping to implement a Dual AMD Epyc solution.

 

Budget (including currency): $10,000 USD (currently about $16,000 AUD)

Country: Australia

Games, programs or workloads that it will be used for: ANSYS Finite Element Analysis and Computational Fluid Dynamics, probably some AI and machine learning, definitely Unity for VR... I get to play with lots of fun toys at work.

Other details (existing parts lists, whether any peripherals are needed, what you're upgrading from, when you're going to buy, what resolution and refresh rate you want to play at, etc):

Hi all, as they used to say, "long time listener first time caller". I am about to get a grant that has in it a nice pot of funds set aside for a workstation build, and this needs to be built from parts, otherwise I am limited to a single supplier, and I am not really interested in their Xeon core workstation options, mostly because they charge rediculus amounts for memory. I am currently looking at Scorptec here in Oz to supply parts, but I am a noob when it comes to Threadripper stuff and have just drooled over videos on YouTube when Linus builds them. For the budget it looks like a 5975WX will be the best choice:

https://www.scorptec.com.au/product/cpu/amd-threadripper/98188-100-100000445wof

The only motherboard that comes up in a search is this:

https://www.scorptec.com.au/product/motherboards/amd-threadripper/88495-pro-ws-wrx80e-sage-se-wifi

It sounds pretty good. I should point out that I am going to need to use a single supplier for everything or the finance team at work will have an issue. Motherboards are my weakness. In the good old days (20ish years ago) I only swapped out things that connect to the motherboard to upgrade (ram, HDD, etc), until a new machine was needed. Recently I built a home office PC with my son, and just got the local computer shop to recommend a motherboard. Watching videos there are so many specs I do not appreciate about motherboards... and although the description of this sounds amazing, it still worries me.
Coolers are also new to me, I mean my son and I put one on the home computer (again recommended by computer store), but for a Threadripper I am concerned. This comes up on Scorptec:

https://www.scorptec.com.au/product/cooling/cpu-coolers-air/75408-mam-d7pn-dwrps-t1

If I can avoid water, that would be amazing, I have never had to deal with water before, and I would rather not damage a major investment like this.
I have looked at is RAM and I am tempted by this, given the cost-effective nature:

https://www.scorptec.com.au/product/memory/ddr4/82426-cmk64gx4m2e3200c16

This gives 256GB across 8 sticks. I have the budget for 512GB using something like this:

https://www.scorptec.com.au/product/memory/ecc-&-registered/97244-ksm32rd4-64hcr

But not sure if it is worth the jump from $1500 AUD to $4800 AUD.
The last thing I have considered is the graphics card, and likely to get an RTX 3090:

https://www.scorptec.com.au/product/graphics-cards/nvidia/85036-zt-a30900d-10p

This will be supplemented by one or more Titan V cards that will need to be sourced second hand (luckily lots of ex crypto mining cards are available). A lot of the engineering that is going to be done on this machine will hopefully be able to use the large double precision core count of the Titan V to accelerate the modelling and simulation. Not sure how nicely these things will play together, so any input about graphics cards would be great.
In addition to that I need storage (not sure the best combination for the motherboard), a case that fits the motherboard, a power supply that will run everything, and anything else I might be forgetting.
All the other stuff like monitors and peripherals are available in abundance at work, nothing special needed.

Edited October 20, 2022 by AerospaceDoctor

[ZetZet]

Is a 32 core Threadripper even worth it now that 7950X exists? Unless you need a shitload of PCIe slots seems like a huge waste of money to me.

 

Or 256GB of RAM I guess.

[brob]

1 hour ago, AerospaceDoctor said:

https://www.scorptec.com.au/product/memory/ddr4/82426-cmk64gx4m2e3200c16

This gives 256GB across 8 sticks. I have the budget for 512GB using something like this:

https://www.scorptec.com.au/product/memory/ecc-&-registered/97244-ksm32rd4-64hcr

But not sure if it is worth the jump from $1500 AUD to $4800 AUD.

 

You first have to decide if ECC memory is important in the use case. If it is non ECC kits shouldn't even be considered. 

 

1 hour ago, AerospaceDoctor said:

If I can avoid water, that would be amazing, I have never had to deal with water before, and I would rather not damage a major investment like this.

 

There are models of the Noctua NH-U14S that would work, see the Noctua compatibility center, https://ncc.noctua.at/.

 

 

[AerospaceDoctor]

1 hour ago, ZetZet said:

Is a 32 core Threadripper even worth it now that 7950X exists? Unless you need a shitload of PCIe slots seems like a huge waste of money to me.

 

Or 256GB of RAM I guess.

Thanks for the reply ZetZet. Do you know if there is a dual 7950 motherboard option, because that could be a nice alternative (I will investigate). What I want for the simulations is to maximise core count because that has a direct impact. There is a linear relationship in most simulation where you double the core count you double performance and hence half the simulation time.

It is also recommended that each core have 8GB of dedicated ram, so that means 32 cores needs 256GB of ram. The same would be true of a dual 7950 build. But there is also a dependence on the size of the model and the amount of ram. So for more than 5 million nodes, 512GB might be best (256GB is good for a million or two).

9 minutes ago, brob said:

 

You first have to decide if ECC memory is important in the use case. If it is non ECC kits shouldn't even be considered. 

 

 

There are models of the Noctua NH-U14S that would work, see the Noctua compatibility center, https://ncc.noctua.at/.

 

 

Thanks brob, the link to the coolers is a great help. I had not considered ECC or not. It looks like it is recommended for workstations with the intended application. That is a great catch nice and early.

[Somerandomtechyboi]

9 minutes ago, AerospaceDoctor said:

Do you know if there is a dual 7950 motherboard option, because that could be a nice alternative

Dual socket consumer is a thing of the past and very niche hence why noone does dual socket consumer anymore, im not even familiar with any of that dual socket or dual slot hardware since thats before im even born and is likely stupid rare

 

Ok so if you need as many cores as possible youd wanna look at a server, if you can get 2 of these epyc 7702 at 3k $ and pair with this dual socket giga mz72 at 900$ Theres your cores, now onto the ram you can buy 16 of These 64gb hynixes for 1tb of ram (512gb per cpu) at 2.2k $, aaand thats already 60% of the damn budget xD though you could buy some 32gb samsungs for only 640$ instead. The cooler just use whatever decent sp3 cooler, the case yea have fun finding one that can fit the board but maybe full towers like the enthoo pro can do it, psu wise depends on how many extra titan v you wanna add you may even need dual psus

[AerospaceDoctor]

Sorry all, I have been busy with work. Thank you for the in for Somerandomtechyboi, my PhD student is very excited about the fact that we could have a 128 cores and 256 threads, with 1TB or ram.

If anyone can help re what we should put this in and power it with, that would be great a great help, completely outside my area of knowledge now!

On an amusing side note, the research team insists on using Lord of the Rings characters for all of the high performance machines, if this goes ahead it will be so much more powerful than all of the others that we have decided to call it Eru, the god of gods.

[TheQuux]

My current workstation build is a SuperMicro H12DSi-N6 with a single EPYC 7453, and I've found that that is brilliant for my needs right now. I could do with the second CPU, but considering that I'm paying for my hardware out of pocket and 7453's aren't cheap, I'll make do for now.

 

Before I move onto the build process, I will say that there are significant tradeoffs to be made with a setup like this. First, it's hot garbage for gaming compared to, e.g., something built around a 5800x: you lose 25% of the single-thread performance, have cache hiccups if your OS moves a thread between CCX's (and worse if it gets moved between sockets). Second, it's missing a lot of creature comforts that you'd expect from a desktop: no audio, *long* boot times (~5 minutes to get to the bootloader), and a crappy virtual graphics card on the BMC that windows occasionally pop up on, making me very confused until I log into the BMC and interact with that window from there.

 

That said, 56 threads make compiles go like a cat on amphetamines. My previous workstation was a 6700k, and a single release build of my software took ~8 minutes. With the EPYC system, it gets through that same build in 45 seconds, and 128G of RAM means that I can run both Chrome *and* Firefox at the same time.

 

So, here's what I've learned about building such a system:

1. Read the manual for your board thoroughly. I tried to skip that the first time, because I've built more than a few systems in my time, going back to socket 754. But no, these server-oriented boards are very weird. In particular, pay attention to the block diagram when you're planning what gets plugged in where.
2. If you go with the dual-socket board (H12DSi, rather than H12SSL), but don't have the second processor installed, you lose the bottom two PCIe slots.
3. The dual socket board isn't that much more than the single-socket board. On the other hand, you'll often pay extra for the SATA-only version. My local pusher (Ahead-IT) charges an extra 50 bucks to not give me SAS on the H12DSi, and you can plug SATA into a SAS slot no problem. The 10G support is up to you; I run fiber at home, so the 10G RJ45 ports were extra cost with no benefit.
4. A dual slot GPU will block one of the other PCIe slots. This isn't actually a problem, because you've got lots of slots and all the x16 slots are actually x16, unlike consumer boards where an x16 slot may actually be x8. OTOH, you'll be sticking in a couple of expansion cards to back-fill stuff that is expected on a workstation but not a server: extra USB ports, audio, ...
5. When you pull the socket cover off of CPU socket 1, put it somewhere safe. This is the only place that the default IPMI password is written.
6. It's critically important where you plug your RAM. If you plug a stick into the wrong slot in a channel, your system may not boot at all, for example. The manual documents that thoroughly though.
7. To get good performance, you need a lot of sticks of RAM. Plan on populating at least 4 per socket, but even then, know that you're losing about half the available memory bandwidth. If you have less than 32 threads per core though, you won't lose that much performance because each core can only use so much bandwidth.
8. Use the BMC console for the initial install, then go through device manager and disable the ASpeed "GPU"; if it's left on, you'll have issues with windows showing up on that virtual display, which will lead to no end of confusion.
9. If you want audio, plan on a USB audio adaptor. Considering the shortage of USB ports on the board, this means that you'll probably want to plan on a USB PCIe card as well. Or you could go with a PCIe audio card, but these tend to be more expensive than they're worth.
10. Without a USB card, you've only got the one internal USB3 connector. I worked around this with the liberal application of copium.

I built mine in a Phanteks Enthoo Primo, which is a delightful case to work in. Lots of room for your hands The Noctua NH-U9 TR4-SP3 fits nicely and keeps everything nice and chill. Considering that the Enthoo Primo isn't even a particularly wide case, you'll probably be fine in anything designed for E-ATX.

 

Now, for your specific requirements, most enterprise IT resellers don't list consumerish stuff like USB cards or empty cases. However, send out some requests for quotes to the resellers in your area and tell them that you're happy to build and configure it all yourself, you just need them to put everything on an invoice, and you're happy to be charged for that. In my experience, resellers will fall over themselves to help you out.

[bassam64i]

Hi AerospaceDoctor, As an engineering professional who uses Finite Element Analysis extensively, along with CAD packages I can tell you the following from experience and research on relevant hardware for FEA/CFD purposes. (I am also in Australia by the way)

 

With regards to the Random Access Memory RAM:

• Channels, channels, channels.... the more, the better. FEA and CFD applications can create a very large data set starting from a few Megabytes all the way to Terabytes. These are iterative calculations that need to run on matrices over and over and over again. This means that very large data sets need to be fetched from RAM and written back to RAM. It doesn't matter how fast your single core performance is, if it it is sitting waiting for a large data set to come through from a dual/quad channel motherboard system. The more channels the better, because if one channel is busy, then another RAM channel can be used, especially in the case of a large data set. By comparison, in gaming, a game binary executables are small and generate a small data set in comparison so that's why in gaming benchmarks you see the number of channels making no difference (3% at best). https://simutechgroup.com/maximizing-memory-performance-for-ansys-simulations/
• If you look at CFD benchmark data, you will see that the fastest computers are the ones that use more memory channels, not the ones that have the fastest single core. The latest 7950X CPU still gets demolished by Threadripper 3rd Gen in CFD/FEA. See OPENFoam CFD results here https://openbenchmarking.org/test/pts/openfoam
• Error Correcting Memory ECC: Memory errors DO HAPPEN, if your CFD/FEA data is loaded in RAM for a long period of time, chances are your data WILL get corrupted at some time. This will result in CFD/FEA calculations being WRONG, or worse, generate a result that will make an engineering design FAIL. For a gaming computer, ECC doesn't matter, it's not critical, if the colour of one pixel was wrong for one gaming frame, who cares. But for FEA/CFD if just one bit in one one number becomes corrupt then it could kill someone due to a failed/wrong analysis result (extreme case here obviously). Veritasium created a fantastic video explaining how this corruption can happen. See video below. Be careful of DDR5 on-die ECC....IT IS NOT the proper SIDEBAND ECC. If you choose DDR5, make sure it is SIDEBAND ECC
• Buy a motherboard that supports >=256GB of RAM. At the moment, gaming motherboards max out at 128GB.... they suck.

With regards to the CPU:

• With regards to the number of cores, CHECK the ANSYS license.... it will say in there the NUMBER of CPUs it supports. Your license might not allow you to use all 128 cores. In which case you would be wasting your money.
• Stick with a single socket motherboard, and buy the core count CPU you desire with a single socket. Dual socket motherboards have a disadvantage when it comes to CPU-to-CPU communication, IT IS SLOW.
• Higher clocks and more cache does help, but memory channel count is also critical, as above.
• Stick with air cooling solutions. They are MORE reliable compared to water pumps when running CFD analysis for hours on end. You can go and watch a movie knowing that your workstation didn't go up in flames with that expensive CPU in it.

With regards to the GPU:

• If you are going to use the GPU to accelerate CFD, then you need a GPU that supports ECC VRAM. This means a QUADRO GPU card. and certainly NOT a gaming card.

Hope this helps. Feel free to reach out

 

[NF-A12x25]

On 10/16/2022 at 9:57 AM, AerospaceDoctor said:

Do you know if there is a dual 7950 motherboard option, because that could be a nice alternative (I will investigate).

no.