256GB ECC Ram + Storage + RAID Card Rec. Needed (Intel)

[CoolaxGaming]

Hey guys!

Memory
 

I am shopping for server memory and I would like some advice. The server we are building needs large amounts of ddr4 memory, the mobo we have has 8 slots, Supermicro MBD-X12SPI-TF-O. 

Here are my constraints
Size: 256GB (Our applications run memory heavy with lighter cpu usage)

Price Points: $2k ($3k is probably our max, not sure at what point performance gained for every extra dollar starts to approach 0)

Configuration: 8x32 or 4x64

ECC
DDR4
Our application benefits from lower latencies more often than higher frequencies. However speeds lower than 2666 is not optimal either, however we acquired this info on various 12x Nanya dimms with 16GB per stick and 2Rx8 So not sure how this will work out. We didnt pick this manufacturer by the way, we were at the mercy of our server provider for that. However this new server will be custom built by us and probably sent to a equinix datacenter, so we have full flexibility on what to buy.

And I was searching exclusively for samsung b die dimms so far, as I have had a good experience with them in the past and been told that b die's are the best. However I am open to other manufacturers if they provide better performance and reliability. This server is the core of our infrastructure and any potential for errors needs to be minimized. 

========================================================================================================================================
Others specs so far:

Xeon 5318N
X12SPI-TF-O
Mellanox ConnectX-6 Dx
3x980 PRO 1 TB (Not bought for this build, from inventory)
2x Kingston KC300 2 TB
Highpoint 4 M.2 NVME RAID Controller (Not bought for this build, can replace)
Everything above is already purchased. Below is what I mgiht get in addition to this

========================================================================================================================================
Storage
Now for storage! Reliability is obviously important, however since most of our applications are pretty light we will probably not need more than 3 TB of space. The server will be writing logs for a ton of applications very quickly. We have two OEM (PM981 1TB) 980 pros running in software raid 0 (Server hasent rebooted in 700 days  its a disaster waiting to happen). We would need something equivelent to that or better. I was thinking 4x 980 PRO's with a raid card but I see sk hynix and others have new nvme ssds that have passed the 980 pro. And since we were restricted by host to this choice, I dont know how well the 980 pro performs for sustained writes. 

SSD Constraints (Very rough):
Min 3 TB Total
RAID Redundency required (I was thinking raid 10?)
Read faster than 2x980 pro's in raid 0
Write faster than 2x 980 pro's in raid 0
Low latency
$2000 budget including the raid card (Does not include anything already purchased)

I have 3x 980 Pro 1TB unopened from a build I was going to do a couple months ago. I was going to use them but like I said, the market seems to have better options now. I was previously set on a p5800x from intel since thats whats in my pc at home and it is a level above everything else, I realized I would not have redundency anymore unless I got 2 of them, which is a bank breaker.
I am also open to having two sets of drives,

OS+Regular App Data Drives
Data wont be written too often besides for updating/deploying new apps. However must be faster than the 2x980 pro in raid 0
High Read speeds required
Low latency required
High Random R+W speeds
Redundency needed

Log Drives
High Sustained Write Speeds (Ram cache will fill up)
High Read speeds in bursts (Reading huge log file)
========================================================================================================================================
Raid Card
Lastly, I need a raid card recommendation for this system as well! I am not sure how the highpoint raid card performs, I got it for my personal pc and never used it. If I decide to go for RAID 10 with 4x 1TB 980 Pros and RAID 0 with 2x KC300 what hardware raid controller would work best? I will only have 1 16x slot open as the mellanox nic and nvidia gpu will take up the other 2.  Would I need 2 raid cards (or would it work just be worse with 1?) for the aformentioned setup? I definetely cant use the Hightop card if I go for 2 sets of drives as it only has 4 slots

Misc:
Chassis and cooling recommendations are welcome! 

Thanks in advance guys!

[manikyath]

for the RAM.. you've already answered your own question:

- 2666MHz memory (from my recent experience shopping for DDR4 ECC.. dont bother looking for higher if you dont need it..)

- if you need expansion space look for 4x64, otherwise just do 8x32.

- for the rest, try to stick as close to QVL list as possible.

 

for the storage:

do you need reliability as in "we cant just go and replace drives" or as in "we need fault tolerance in the array"? because that's a very different setup..

 

as for the raid card:

do you *need* it to be a hardware raid solution? if you want to run M.2 drives i'm more thinking towards a bifurcation card that just sticks 4 m.2 slots right onto the PCIe lanes, and let software do the rest. that way you have a lot more flexibility, and a lot more control over your array.

 

as for the writes you mention:

do you have figures on what sort of storage bandwidth you need, and how much data you'll be writing to the drives? if you have these numbers, it's just math to determine how long a specific drive is 'rated' to last in the conditions you set up.

[Electronics Wizardy]

For storage one other option is 2.5in ssds. How about something like a Micron 7400 MAX. Should be able to get 2 of those for sub 2k easily, and then you can have it hot swap too.

 

These drives generlly do very well with sustained performance.

 

54 minutes ago, CoolaxGaming said:

OS+Regular App Data Drives
Data wont be written too often besides for updating/deploying new apps. However must be faster than the 2x980 pro in raid 0
High Read speeds required
Low latency required
High Random R+W speeds
Redundency needed

re you running vms? Id normally go with pair of slow raid 1 drives for boot and os, then those 2.5in ssds for. dat, and  log(maybe get 4 of em). How big do you need your log drive?

 

55 minutes ago, CoolaxGaming said:

Chassis and cooling recommendations are welcome! 

Id really suggest getting the supermicro case built for this board, or getting the dell/hpe/lenovo eqv if you can. There much nicer hardware to work with.

 

56 minutes ago, CoolaxGaming said:

Raid Card
Lastly, I need a raid card recommendation for this system as well! I am not sure how the highpoint raid card performs, I got it for my personal pc and never used it. If I decide to go for RAID 10 with 4x 1TB 980 Pros and RAID 0 with 2x KC300 what hardware raid controller would work best? I will only have 1 16x slot open as the mellanox nic and nvidia gpu will take up the other 2.  Would I need 2 raid cards (or would it work just be worse with 1?) for the aformentioned setup? I definetely cant use the Hightop card if I go for 2 sets of drives as it only has 4 slots

THere really isn't  hardware raid card for nvme drives that I've seen. They all seem to be pcie switches using software raid. So Id just get the switches and use standard software raid like zfs or storage spaces.