NAS build plan and ECC CPU question

[thorgim]

Hi LTT community, I'm excited to join the forum after gaining so much insight from the LTT videos on YouTube during my research phase in preparation for my next build.

 

Brief background, feel free to skip this part: I'm currently running a dated HP Proliant Micro Server (the very first version), with 4x2TB drives in RAID-Z1, 8GB of 800Mhz ECC RAM and FreeNAS. This has already led me to work with external hard drives for my video files, because the HP's specs say it can handle max 4 x 2 TB, which are full.

 

Goal

Build a home server that is as energy efficient and silent as possible, mainly functions as a NAS (to edit 4k videos from, store backups and serve my media to the LAN), potentially host some light weight containerized services for home automation and if I accidentally end up with enough juice also a Windows VM (not a priority at all).

 

Network

10G Ethernet with the "MikroTik CRS305-1G-4S+IN" switch and "Connectx-2 PCIe SFP+" cards as well as matching 10G SFP+ DAC cables for the short distance connections I need to cover.

 

NAS Hardware

Case: Fractal Design Define R5 Black

HDD: 3x 12TB WD UltraStar SATA (ZFS, RAID-Z1)

Host Bus Adapter (HBA): LSI 9211-8i from eBay

Power Supply: Fractal Design ION+ 560P

RAM: 16GB ECC ???

Mainboard: ???

CPU: ???

 

Still open is the choice of CPU, mainboard and RAM, because I'd like to go with ECC RAM but was super confused with the compatibility and availability with low cost CPUs. I'm especially interested in suggestions about these components.

 

I was inspired by LTT's recent NAS build video about going Ryzen because of their unofficial ECC RAM support, power efficiency and price, but got lost when trying to find a matching mainboard and compatible RAM. The only solution I found so far is an "AMD RYZEN 5 2600", "ASUS ROG Strix B450-F" with "16GB Kingston Server Premier RAM (KSM26ED8/16ME)", which seems to be the only ECC RAM listed by ASUS in their supported hardware PDF. However, this feels like overkill for my use case.

 

I'm also considering to get a used workstation with a Xenon and ECC RAM, which might bring down the cost because it gets me a case, mainboard, CPU, power supply and potentially RAM.

 

As a third alternative I consider an "i3-8100T" that apparently supports ECC RAM with a microtec board, if I can find one that allows me to use my two PCIe x8 cards (10G NIC and HBA).

 

What do you think about the options that I'm considering, do you have a recommendation in this scope or am I even missing a completely different angle?

[Windows7ge]

ASRock Rack has a couple of workstation/server type boards for AMD with wider ECC support.

If you want to go Intel you have broader options in the Xeon market from ASRock Rack and Supermicro.

 

If you're really on a budget look into retired servers on eBay. There's some really good deals there.

[Electronics Wizardy]

15 minutes ago, thorgim said:

Host Bus Adapter (HBA): LSI 9211-8i from eBay

You should have enough onboard sata to not need this.

 

16 minutes ago, thorgim said:

What do you think about the options that I'm considering, do you have a recommendation in this scope or am I even missing a completely different angle?

Because you don't need much cpu power or ram here, id get a used supermicro boards on ebay, the 1155 ones are pretty cheap now, then put a old e3 1230 v3 or whats compatitble with your board, and some ddr3 udimms.

[Kilrah]

Retired servers are likely not to satisfy the "as energy efficient and silent as possible" requirement though...

[thorgim]

The energy efficiency and noise issues were also something that made me sceptical about the used server market. Noise can probably be dealt with through exchanging fans and coolers but these Xeon server CPUs seem to be crazy power hungry looking at their TDP, or does it look worse than the actual mostly idling CPU will be effectively?

 

EDIT: The Intel Xeon E-2278GEL doesn't actually look to bad in terms of TDP.

EDIT2: But is expensive

EDIT3: Maybe the Intel Xeon E3-1240L v5 instead

[thorgim]

18 minutes ago, Electronics Wizardy said:

35 minutes ago, thorgim said:

Host Bus Adapter (HBA): LSI 9211-8i from eBay

You should have enough onboard sata to not need this.

So the onboard SATA controller is not expected to be worse than a dedicated PCIe HBA card?

[Electronics Wizardy]

29 minutes ago, thorgim said:

So the onboard SATA controller is not expected to be worse than a dedicated PCIe HBA card?

They will work the same for your use case. You really only need the hba if you need more ports.

 

41 minutes ago, thorgim said:

The energy efficiency and noise issues were also something that made me sceptical about the used server market. Noise can probably be dealt with through exchanging fans and coolers but these Xeon server CPUs seem to be crazy power hungry looking at their TDP, or does it look worse than the actual mostly idling CPU will be effectively?

 

EDIT: The Intel Xeon E-2278GEL doesn't actually look to bad in terms of TDP.

EDIT2: But is expensive

EDIT3: Maybe the Intel Xeon E3-1240L v5 instead

TDP doesn't matter for idle power consumption, idle power does. Dual xeon s are pretty hungry idle, 100w is about as low as you can get many of them, and it goes up with nics and ram.

 

 

[thorgim]

Thanks for clarifying the TDP issue. For building with new hardware I'm now looking at these options:

 

Mainboard: ASRock Rack C236M WS (thanks @Windows7ge, that's a nice looking board)

CPU: Intel Xeon E3-1240 v5 or E3-1230 v5
RAM: Transcend DIMM 8GB, DDR4-2133, CL15-15-15, ECC (TS1GLH72V1H)

NIC: Connectx-2 PCIe SFP+

Idle: <40W (here they say only ~55 W max under load)

Cost: 530€

 

Mainboard: Supermicro X11SDV-4C-TLN2F (with Xeon Processor D-2123IT and 10G NIC)

RAM: Kingston Server Premier DIMM 8GB, DDR4-2400, CL17-17-17, ECC

Idle: ~50W

Cost: 600€

 

Mainboard: Supermicro X10SDV-4C-TLN4F (with Xeon Processor D-1518 and 10G NIC)
RAM: Transcend DIMM 8GB, DDR4-2133, CL15-15-15, ECC (TS1GLH72V1H)

Idle: ~20W

Cost: 610€

 

Given that I want to spend the money, the last option sounds like the most energy efficient (and silent, given the passive cooling). I guess buying a used system won't get me anywhere close in terms of power consumption. What do you think?

[mariushm]

There's a company selling 300 quanta servers: https://www.webhostingtalk.com/showthread.php?t=1799745

 

Quote

1U Compute
50x D51PC-1U (v3/v4) w/ onboard RAID, Dual Heatsink, 2x 10G on mezz, 2x 10G PCIE (DAS2ETH38D0), 10x SFF Trays, Dual PSU, Rails - $250ea

2U
50x D51B-2U (v3/v4) w/ 9271-8I, 2x 10G on mezz, Dual Heatsink, 24x SFF Trays, Dual PSU, Rails - $250ea
50x S210-X22RQ (v1/v2) w/ 9271-8I, 2x 10G on mezz, Dual Heatsinks, 24x SFF Trays, Dual PSU, Rails - $150ea

Storage
150x SD1Q-1ULH 12x3.5” SATA/SAS + 6x2.5” (SSD only) + 1x PCIe 2280 M.2 w/ D-1541 CPU Onboard, 2x 10G on mezz, Trays, Dual PSU, Rails - $450ea

Pictures available upon request!
All CPU/memory options in stock to custom configure. All systems have 1 Year warranty – can ship globally (currently in Minnesota). Free shipping in continental US. Please send DM or email me.

judd dot L @ core4solutions dot com

d-1541 is 8 core / 16 threads 45w TDP cpu : https://ark.intel.com/content/www/us/en/ark/products/91199/intel-xeon-processor-d-1541-12m-cache-2-10-ghz.html

 

Here's the system specs: https://www.qct.io/product/index/Storage/Storage-Server/1U-Storage-Server/QuantaGrid-SD1Q-1ULH#specifications

 

If you're outside US, shipping's probably gonna be something like 200$ though.

 

oh yeah... and another downside.. being 1u it's probably gonna be a bit noisy, but if you put it in a rack somewhere who cares...

Maybe get a 150$ server also if you have the money, maybe make a 24 SSD NAS in the future when they get super cheap, or maybe you can reuse some laptop hard drives.

[thorgim]

46 minutes ago, mariushm said:

f you're outside US, shipping's probably gonna be something like 200$ though.

I contacted them anyway to find out. Thanks for pointing me to that option, although it's probably rather noisy (unfortunately I don't have a sufficiently well networked dark closet to stow this away in).

[mariushm]

@thorgim email him ... the guy doesn't work for Quanta, these are servers pulled from a datacenter... they may have some UPS/DHL contract and ship thinigs relatively cheap to you.

 

The 2U servers at 250$ each would be less noisy I imagine....

 

But even the 1u servers could be not so much noisy, it may spin the fans really fast at boot and then slow down the fans once the temperature is detected. Or, there may be fan control options in bios or through the remote. 

The servers use 400w platinum efficiency power supply so they don't make lots of heat, and the cpu being 45w tdp should also be fairly cold especially if it idles most of the time... the most heat would be produced by the mechanical drives and if you don't fill all 12 bays from the start it would probably be a cold machine.

 

You could also possibly customize them to replace the fans with 80-120mm ones ... just convert the machine into a 3U or 4U machine ... cut the metal from the top and make your own custom cover and replace the fans with your own.

 

Maybe email the guy and ask if the fans are always spinning fast or they slow down after a while or there's options for "silent mode" ?

[thorgim]

@mariushm, with import taxes and shipping the offer isn't as convincing anymore. But thanks a lot for the recommendation, for folks in the states this sure is a sweet deal.

[thorgim]

2 hours ago, thorgim said:

I'm now looking at these options:

and also potentially the following Xeon Bronze based setup, which is worse in terms of single core performance but should be competitive in a multi tasking context.

 

CPU: Xeon Bronze 3104

Mainboard: Supermicro X11SPM-TPF (incl. 10G SFP+ NIC)

RAM: Kingston Server Premier DIMM 8GB, DDR4-2666, CL19-19-19, ECC

Idle: ??? W

Cost: 640€

 

Even though it's a bit pricier it is for sure more upgradable (probably unreasonably upgradable ;))

[mariushm]

You lose the remote management and 10gbps onboard, but a 10g sfp+ network card costs as little as 20-30$

 

In the 500-600 euro / pounds / etc, you can buy new .. Here's a configuration.

You get an 8 core threadripper and you can upgrade up quite a bit in the future... also you can probably get these 1900x cpus used much cheaper

You'll want at least a pair of 8 GB sticks, ideally 4 sticks because threadripper knows quad channel. though for nas doesn't matter that much... so a pair is probably enough.

The case... you may want a case with loads of internal 3.5" bays, but the one below also has some 5.25" bays where you could install one of those adapter things, which converts 2-3 5.25" bays into 4-5 x 3.5" bays.

The beauty of the threadripper is that it has 64 pci-e lanes (60 +4 to chipset), so you have 3 m.2 bays full pci-e 3.0 x4  and you also have 3 pci-e x16 slots ... put a cheap video card, a dual 10g card and a hba controller and youre good.

For around 30$/euro/pounds/whatever more you can get motherboard with 5 pci-e x16 slots : https://uk.pcpartpicker.com/product/kydxFT/gigabyte-x399-aorus-pro-atx-tr4-motherboard-x399-aorus-pro

Either way, you have 8 sata ports on the board already.

Both suggested boards are plain ATX so they'll work in regular ATX cases.

PCPartPicker Part List

Type	Item	Price
CPU	AMD Threadripper 1900X 3.8 GHz 8-Core Processor	£143.40 @ Alza
CPU Cooler	Noctua NH-U12S TR4-SP3 54.97 CFM CPU Cooler	£61.30 @ Amazon UK
Motherboard	ASRock X399 Phantom Gaming 6 ATX sTR4 Motherboard	£225.40 @ Alza
Memory	Kingston 8 GB (1 x 8 GB) DDR4-2666 Memory	£47.97 @ Currys PC World Business
Case	Deepcool TESSERACT BF ATX Mid Tower Case	£35.47 @ Scan.co.uk
Power Supply	Antec Earthwatts Gold Pro 750 W 80+ Gold Certified Semi-modular ATX Power Supply	£80.41 @ Amazon UK
	Prices include shipping, taxes, rebates, and discounts
	Total	£593.95
	Generated by PCPartPicker 2020-03-12 20:01 GMT+0000

 

[thorgim]

Thanks, this seems to be more performance for the money, indeed.

Currently, the price / power consumption sweet spots seems to be with these two setup I described above, what do you think?

3 hours ago, thorgim said:

Mainboard: ASRock Rack C236M WS (thanks @Windows7ge, that's a nice looking board)

CPU: Intel Xeon E3-1240 v5 or E3-1230 v5
RAM: Transcend DIMM 8GB, DDR4-2133, CL15-15-15, ECC (TS1GLH72V1H)

NIC: Connectx-2 PCIe SFP+

Idle: <40W (here they say only ~55 W max under load)

Cost: 530€

 

[...]

 

Mainboard: Supermicro X10SDV-4C-TLN4F (with Xeon Processor D-1518 and 10G NIC)
RAM: Transcend DIMM 8GB, DDR4-2133, CL15-15-15, ECC (TS1GLH72V1H)

Idle: ~20W

Cost: 610€

I also looked into Supermicro A2SDI-H-TF (with Atom C3758 and 10G Base-T NIC) as an alternative to that in terms of low power consumption, but am so far not convinced that the cut in performance and increase in price actually provides a significant reduction in power consumption.

 

EDIT: Actually, let me update the Intel Xeon E3-1200 series v5 CPU with the Kaby Lake v6 version, which is similarly priced and more energy efficient.

[thorgim]

End of day favourite system (I'm going to sleep over it and look forward to you proving me wrong, showing me something that is competitive in terms of price / energy efficiency ratio and low noise):

 

CPU: Intel Xeon E3-1220 v6 (~230€)

Mainboard: ASRock Rack C236M WS (~200€)

RAM: Transcend DIMM 8GB DDR4-2133 (~60€)

PSU: Seasonic Platinum Series Fanless 400W (~120€)

NIC: Connectx-2 PCIe SFP+ (~50€)

HDD: 3x WD UltraStar 12TB (ZFS, RAID-Z1, ~1050€)

Case: Fractal Design Define R5 Black (~100€)

Cost: ~1800€

 

The used budget alternative of that setup with higher energy consumption would be with a Intel Xeon E3-1230 v3 and Supermicro X10SLL-F for ~1500€.

 

The closest competition in terms of ready built NAS systems should be the QNAP TS-832X-8G, right?

[thorgim]

Just for the record the setup of my previous post is the one I went for and it's OP but I like it, HDDs seem to make the largest difference in power consumption.