Server build feedback

[tdallas]

I know this is a bit niche. I'm not looking for gaming. I'm looking for long-term durability and compute power (less in terms of speed and more in terms RAM capacity). This is a silver-spoon build and I want to make sure I get it right. Any thoughts or suggestions are appreciated.

 

## Budget and location

I am a researcher at a US university who has received funding that will involve building a server to use for some teaching and compute workloads. I have funding to spend up to around $11000. Before folks tell me to use campus HPC, I have a job that's been pending for a week taking a single node. The resources here have to support lab tasks as well as 20+ users doing light-ish analyses (~16gb RAM to each, small storage demands).

## Aim
I care less about GPU and immediate speed and more about long-term durability and overall capability (especially in terms of RAM, as we do some work that is larger data and languages that hold all objects in memory). It will likely be used for some light video editing work and some image segmentation tasks, but nothing that would really tax a GPU too much. Folks that say I should just cloud services for this. I agree, but this money needs to be spent out in a year or it goes away. This will be my first enterprise build, and certainly my most expensive build, so I'm hoping to get feedback on 1) if this thing as listed will even post (there is no pcpartpicker for server stuff really) and 2) if the mash of components makes sense or if I'm throttling myself somewhere.

 

## Monitors
Ideally zero.

## Peripherals
Ideally zero.

## Why are you upgrading?
This is a start build. We have 3 other high(er) memory machines in the lab, but they are not enterprise parts so I question long-term viability. They are also capped at around 126 gb RAM. I want this build to be extensible (16 RAM channels on the board helps this, but I guess I'm limited on which CPUs I can run given the board).

 

## Planned build as of now

Mobo: SUPERMICRO MBD-H12DSi-N6-O Extended ATX Server Motherboard Socket SP3

CPU: (2 of them) AMD EPYC 7763 Milan 2.45 GHz Socket SP3 280W

RAM: (starting with 6?) Supermicro MEM-DR464L-HL03-ER32 64Gb 2Rx4 PC4-25600 Ecc Memory

GPU: MSI GeForce GTX 1080 Ti 11GB GDDR5X (chosen based on forums saying the selected board won't support some of the other options like 4080 etc.)

SSD: SAMSUNG 980 PRO SSD 1TB, PCIe 4.0 M.2 2280

PSU: be quiet! Dark Power 13 1000W ATX 3.0

CPU cooler: (2, if they'll fit but probably overkill) Noctua NH-D15

Case: Antec Performance 1 FT, RTX 40

Then case fans as needed and probably a couple 1TB HDD I have.

*Total estimated cost*: ~7500 US

 

[brob]

Use https://ncc.noctua.at/ to check for Noctua cooler compatibility.

 

Are the reasons for newer Nvidia GPU not working due to hardware, driver or o/s issues?

 

Get a much larger NVMe drive, 4TB are relatively inexpensive at the moment. 

 

Consult the motherboard user manual for valid CPU / memory configurations. Some possible configurations are not recommended.

 

[Blue4130]

8 hours ago, tdallas said:

I know this is a bit niche. I'm not looking for gaming. I'm looking for long-term durability and compute power (less in terms of speed and more in terms RAM capacity). This is a silver-spoon build and I want to make sure I get it right. Any thoughts or suggestions are appreciated.

 

## Budget and location

I am a researcher at a US university who has received funding that will involve building a server to use for some teaching and compute workloads. I have funding to spend up to around $11000. Before folks tell me to use campus HPC, I have a job that's been pending for a week taking a single node. The resources here have to support lab tasks as well as 20+ users doing light-ish analyses (~16gb RAM to each, small storage demands).

## Aim
I care less about GPU and immediate speed and more about long-term durability and overall capability (especially in terms of RAM, as we do some work that is larger data and languages that hold all objects in memory). It will likely be used for some light video editing work and some image segmentation tasks, but nothing that would really tax a GPU too much. Folks that say I should just cloud services for this. I agree, but this money needs to be spent out in a year or it goes away. This will be my first enterprise build, and certainly my most expensive build, so I'm hoping to get feedback on 1) if this thing as listed will even post (there is no pcpartpicker for server stuff really) and 2) if the mash of components makes sense or if I'm throttling myself somewhere.

 

## Monitors
Ideally zero.

## Peripherals
Ideally zero.

## Why are you upgrading?
This is a start build. We have 3 other high(er) memory machines in the lab, but they are not enterprise parts so I question long-term viability. They are also capped at around 126 gb RAM. I want this build to be extensible (16 RAM channels on the board helps this, but I guess I'm limited on which CPUs I can run given the board).

 

## Planned build as of now

Mobo: SUPERMICRO MBD-H12DSi-N6-O Extended ATX Server Motherboard Socket SP3

CPU: (2 of them) AMD EPYC 7763 Milan 2.45 GHz Socket SP3 280W

RAM: (starting with 6?) Supermicro MEM-DR464L-HL03-ER32 64Gb 2Rx4 PC4-25600 Ecc Memory

GPU: MSI GeForce GTX 1080 Ti 11GB GDDR5X (chosen based on forums saying the selected board won't support some of the other options like 4080 etc.)

SSD: SAMSUNG 980 PRO SSD 1TB, PCIe 4.0 M.2 2280

PSU: be quiet! Dark Power 13 1000W ATX 3.0

CPU cooler: (2, if they'll fit but probably overkill) Noctua NH-D15

Case: Antec Performance 1 FT, RTX 40

Then case fans as needed and probably a couple 1TB HDD I have.

*Total estimated cost*: ~7500 US

 

Is there a reason why you are going with last gen epyc as opposed to current gen?

[tdallas]

18 hours ago, brob said:

Use https://ncc.noctua.at/ to check for Noctua cooler compatibility.

 

Are the reasons for newer Nvidia GPU not working due to hardware, driver or o/s issues?

 

Get a much larger NVMe drive, 4TB are relatively inexpensive at the moment. 

 

Consult the motherboard user manual for valid CPU / memory configurations. Some possible configurations are not recommended.

 

Nice. Thanks!

 

I'm on shakiest ground with GPU issues. Couldn't find much, and am basing it (perhaps wrongly) on a similar board (https://forums.servethehome.com/index.php?threads/supermicro-epyc-motherboards-list-of-graphics-cards-that-work-i-need-help-to-make-a-list.34935/)

 

Will get larger NVMe drive. I'm building it so less focused on storage relative to compute power, as I need more compute than storage, but 4TB would likely help with longer-term viability (I want this server to be in the lab for 10 years *fingers crossed*).

 

I think I got the right CPU/RAM config worked given supermicro compatibility checker, but I will definitely be triple checking.

[tdallas]

17 hours ago, Blue4130 said:

Is there a reason why you are going with last gen epyc as opposed to current gen?

Not especially. I sadly didn't even notice that. The price point on the CPUs is rough ($2300 per CPU for a dual socket build means most of the cost of the build is going into CPUs). But I can look at using the 7003 series instead of 7002 (if I'm getting that right?). Board supports those two series of Epyc. Thanks!

[Blue4130]

9 hours ago, tdallas said:

Not especially. I sadly didn't even notice that. The price point on the CPUs is rough ($2300 per CPU for a dual socket build means most of the cost of the build is going into CPUs). But I can look at using the 7003 series instead of 7002 (if I'm getting that right?). Board supports those two series of Epyc. Thanks!

Using 7003 IS last gen. Going with 7002 would be 2 gens old. 9000 is the current generation. I am not going to disuade you from going 7003, it's still an epic (epyc) system to build on and you can carry your dollar even further, just want to make sure you are aware that you will be buying into an older platform with the associated tech such as ddr4 and pcie 4.0 rather than the newer ddr5 and pcie 5.

 

Going 9000 series will obviously cost more, bu it has a roughly 17% performance increase over 7003 series.

[tdallas]

On 12/8/2023 at 10:57 PM, Blue4130 said:

Using 7003 IS last gen. Going with 7002 would be 2 gens old. 9000 is the current generation. I am not going to disuade you from going 7003, it's still an epic (epyc) system to build on and you can carry your dollar even further, just want to make sure you are aware that you will be buying into an older platform with the associated tech such as ddr4 and pcie 4.0 rather than the newer ddr5 and pcie 5.

 

Going 9000 series will obviously cost more, bu it has a roughly 17% performance increase over 7003 series.

Ah nice. That makes sense. I think I'm limited to the 7002/7003 series based on the mobo (https://www.newegg.com/supermicro-mbd-h12dsi-n6-o-supports-amd-epyc-7003-7002-series-processors/p/N82E16813183795?Item=N82E16813183795). It's also nice to use some spare server RAM I have sitting around that is ddr4.

 

Thanks for your help.

[brob]

37 minutes ago, tdallas said:

Ah nice. That makes sense. I think I'm limited to the 7002/7003 series based on the mobo (https://www.newegg.com/supermicro-mbd-h12dsi-n6-o-supports-amd-epyc-7003-7002-series-processors/p/N82E16813183795?Item=N82E16813183795). It's also nice to use some spare server RAM I have sitting around that is ddr4.

 

Thanks for your help.

 

Are other motherboards not an option?

 

[tdallas]

4 hours ago, brob said:

 

Are other motherboards not an option?

 

Happy to consider other motherboards (still would like them to have ddr4 RAM). This one just had a decent price point, supported dual CPU, and was from a brand that I recognize as making enterprise quality gear. The e-ATX size of the thing makes me think that this tower is going to be pretty chonky, but we have space in the lab.

[brob]

4 hours ago, tdallas said:

Happy to consider other motherboards (still would like them to have ddr4 RAM). This one just had a decent price point, supported dual CPU, and was from a brand that I recognize as making enterprise quality gear. The e-ATX size of the thing makes me think that this tower is going to be pretty chonky, but we have space in the lab.

 

What is the reason for sticking with DDR4?

[tdallas]

15 hours ago, brob said:

 

What is the reason for sticking with DDR4?

I have 4 decently quick (3200 RDIMM) RAM modules sitting around that should be used. The speed increase of DDR5 is not a huge concern for the work we're doing as well, as most of the computing will be more core-limited and isn't super time or i/o sensitive.

[brob]

1 minute ago, tdallas said:

I have 4 decently quick (3200 RDIMM) RAM modules sitting around that should be used. The speed increase of DDR5 is not a huge concern for the work we're doing as well, as most of the computing will be more core-limited and isn't super time or i/o sensitive.

 

Staying with DDR4 may lead to higher costs over the long term as supply dwindles.

 

DDR5 offers on die ECC which should contribute to more reliable memory. It will also be available in higher capacity modules.

 

[Blue4130]

6 hours ago, tdallas said:

I have 4 decently quick (3200 RDIMM) RAM modules sitting around that should be used. The speed increase of DDR5 is not a huge concern for the work we're doing as well, as most of the computing will be more core-limited and isn't super time or i/o sensitive.

Are they registered ecc?
Of course they are. I need to slow down and read better.