Mini-ITX ECC Intel MicroServer build help

[Fuzebox]

Hey guys. I just procured a HP Proliant H54L MicroServer. It's old, from 2013, and while it works fine, it got me itchin' to do a more serious build.

 

I will be running Ansible-NAS on Ubuntu Server 20.10 (Arch isn't really a server os, tough I use Arch,. btw.

I prefer rolling release and pacman over debian based and apt. But there are advantages of Ubuntu. CentOS was on the cards, but i don't want every other package to be outdated. Suse is a good alternative as well. FreeNAS is great and all but FreeBSD has some compatibility issues for me in this use case. Mainly good docker support. I am also a 20+ year linux user so id rather stick with what i know very well.

 

Here are my criteria:

• It needs to have ECC - so no AMD. I will be running ZFS - and while there are wild discussions all over the place on wheter ECC is needed, i'd rather air on the safe side.
• It needs to be Mini-ITX, as i am putting it in a very small case - i'll link 2-3 alternatives further down.
• It needs to support at least 4+1 drives. Say 4 SATA or a SASx4 at least, and a M.2. Or say 6 SATA or SAS so i can put the OS on an SSD.
• I would prefer 10G networking, tough id settle for 5, 2.5 or even 1GBx2, as i can put a 10G in later, unless i will need a SAS card... (I can get 10Gbps internet here in Sweden so might as well future proof) Im considering turning it into a router also. Hence more than one network port.
• I will be running several docker image services and possibly a VM so a low spec Xeon should suffice, tough if a non WS/Server CPU is supported and can handle it that would do fine. Needs ECC though so, probably Xeon, yes?

 

Here are some of the components I'm considering:

Case:

• Inter-Tech SC-4100 or Inter-Tech SC-4004 or SuperChassis 721TQ-250B / SuperChassis 721TQ-250B2 - i can tell the difference between the Super Micro cases. Airflow seems very restricted however.

Any other cases you know of that has 4x front drive bays, mitx, small and would fit the bill?

 

Motherboard:

• ASUS P11C-I Interesting... But i have questions about the integrated SAS. Only 1 SATA and one M.2 OR SATA? Does the SAS require me to use RAID. I want separate block devices and run ZFS. Does the SAS controller have IT mode, or can i get separate raw block devices some other way? Data sheet is unclear.
• Gigabyte C246N-WU2 3-4 SATA, 1 miniSASx4. 1 M.2. 2 Gbit lan. Optane. SATA-DOM - works the same as SATA for this application? Ie 3 hdds on sata, one of them on sata-dom.
• Asrock RACK E3C236D2I Now this is interesting. Seems to fit the bill rather well? Also validated for Linux. 2xGbit Lan. Wake on lan etc.
• There are lots of Asrock RACK boards that vary in specs. 2x10G, amount of SATA, SAS etc. Any ideas?
• SuperMicro Boards seem limited in what i require. Found one option but nah. If you know of one please share.

 

CPU and RAM ill choose based on the motherboard i land on.

 

Now for some questions for you fine folks:

 

• The cases im looking at appear to have a max CPU cooler height of 43mm. I was hoping to use a Noctua low profile cooler, but they are about 60mm. Can you think of a cooler that will work in this case?
• Using SAS - are there any issues with getting raw block devices in the OS and running ZFS unencumbered? Do i have to use raid with SAS? What is IT mode - really?
• SATA-DOM - any issues plugging it in to a drive just as the others?
• What are your taughts on this setup?
• Any other issues you can think of cropping up with this project?

 

Should i just go for a slightly larger case for better cooler compatibility? I really like the idea of a micro server tough!

 

I look forward to hearing your input.

 

[porina]

Presuming you have the N54L microserver? I'm still using one of those right now, although I don't need CPU power so it still does the job.

 

Along similar lines, look at the gen 8 microserver. These have Skylake-era CPU support. Base models with a Celeron are pretty cheap used. They can be upgraded to an i3 or Xeon, if you can find the right models at low power usage to fit in one (yes, some i3 CPUs officially support ECC). Other features can be added with the PCIe slots. In mine I have a 2.5G ethernet card and a GPU. The Gen 8 has the 4 drive bays, as well as one internal SATA connector you can use for a boot drive for example. From memory you'll need to make or source a power cable as the only spare one is 3.5" floppy style.

[Fuzebox]

1 minute ago, porina said:

Presuming you have the N54L microserver? I'm still using one of those right now, although I don't need CPU power so it still does the job.

 

Along similar lines, look at the gen 8 microserver. These have Skylake-era CPU support. Base models with a Celeron are pretty cheap used. They can be upgraded to an i3 or Xeon, if you can find the right models at low power usage to fit in one (yes, some i3 CPUs officially support ECC). Other features can be added with the PCIe slots. In mine I have a 2.5G ethernet card and a GPU. The Gen 8 has the 4 drive bays, as well as one internal SATA connector you can use for a boot drive for example. From memory you'll need to make or source a power cable as the only spare one is 3.5" floppy style.

I have looked at the never generations. They are nice and all, and warranty is good too. But i will have to run raid, unless i get a SAS HBA with IT support to run ZFS - afaik. They also are proprietary spec, and i am a open source enthusiast. I will probably even flash a open source bios, cant do that on those. There are other issues with them as well. Too constrained for my tastes.

Also the N54L can be flashed with a modded bios to enable SATA3gb and the use of a SSD on the internal connector and more.

[porina]

8 minutes ago, Fuzebox said:

They also are proprietary spec, and i am a open source enthusiast. I will probably even flash a open source bios, cant do that on those. There are other issues with them as well. Too constrained for my tastes.

Fair enough. Coincidentally, I'm also considering a low cost ECC solution because in my case 4 storage drives are insufficient, so will follow your thread for ideas. Also I found the real "server stuff" in later gens (not present in N54L) to be annoying as I don't want or need it.

 

11 minutes ago, Fuzebox said:

Also the N54L can be flashed with a modded bios to enable SATA3gb and the use of a SSD on the internal connector and more.

I'd have to look this up out of interest.

[Fuzebox]

There are some interesting options out there for home server use. I may kick my self a year from now choosing to go for a microITX micro server build. Whatever, i can get another case at a later time.

You could look at https://www.silverstonetek.com/product.php?pid=452&area=en It supports 12 drives. There are many tower cases with hot swap bays as well ofc.

[svmlegacy]

FYI, Ryzen CPU's support ECC.

https://www.asrockrack.com/general/productdetail.asp?Model=X570D4I-2T#Specifications

[Fuzebox]

Just now, svmlegacy said:

FYI, Ryzen CPU's support ECC.

https://www.asrockrack.com/general/productdetail.asp?Model=X570D4I-2T#Specifications

Yes i know, but they are not certified for it. They are not server spec processors and their ECC support is being questioned in some circles. I would go AMD, but their server spec CPUs are expensive and or a bit too powerful.

Now i have not read up about ZFS on AMD with ECC vs Intel. If it doesn't matter i might go AMD. If anyone knows more...

[Electronics Wizardy]

1 hour ago, Fuzebox said:

Yes i know, but they are not certified for it. They are not server spec processors and their ECC support is being questioned in some circles. I would go AMD, but their server spec CPUs are expensive and or a bit too powerful.

Now i have not read up about ZFS on AMD with ECC vs Intel. If it doesn't matter i might go AMD. If anyone knows more...

https://www.newegg.com/asrock-rack-x570d4i-2t-amd-ryzen-3rd-generation-series-processors/p/N82E16813140045

How about this guy 

 

Ecc support, impi, intel 10gbe, 4 sodimm slots.

 

I have the x470 version of this board and it works well, this would be my pick here.

 

Or go xeon d/epyc 3000. 

 

https://www.newegg.com/supermicro-mbd-x10sdv-4c-tln4f/p/1JW-0006-00016

 

this guy only have a quad core, but 4 full size dimms with rdimm support(all of the boards you listed are udimm only), impi, 10gbe.

[Fuzebox]

1 minute ago, Electronics Wizardy said:

https://www.newegg.com/asrock-rack-x570d4i-2t-amd-ryzen-3rd-generation-series-processors/p/N82E16813140045

How about this guy 

 

Ecc support, impi, intel 10gbe, 4 sodimm slots.

 

I have the x470 version of this board and it works well, this would be my pick here.

 

Or go xeon d/epyc 3000. 

 

https://www.newegg.com/supermicro-mbd-x10sdv-4c-tln4f/p/1JW-0006-00016

 

this guy only have a quad core, but 4 full size dimms with rdimm support(all of the boards you listed are udimm only), impi, 10gbe.

I looked at those. Gonna take another glance tomorrow. Its the middle of the night here.

Turns out however - according to ansible-nas creator at least - ECC isn't really necessary for ZFS unless you run mission critical enterprise stuff. I really don't mind one flipped bit in a trillion when most files are video, that can handle that anyways. Gonna look further into that, but if true quite a few more board options just opened up.

 

[Electronics Wizardy]

2 minutes ago, Fuzebox said:

I looked at those. Gonna take another glance tomorrow. Its the middle of the night here.

Turns out however - according to ansible-nas creator at least - ECC isn't really necessary for ZFS unless you run mission critical enterprise stuff. I really don't mind one flipped bit in a trillion when most files are video, that can handle that anyways. Gonna look further into that, but if true quite a few more board options just opened up.

 

Yea for. home server ecc is probably overkill, id probably just get a b460 or h410 board here and a i3 10100. That money would probably be better spent on backups or other things.

 

How much do you want 10gbe, and which standard?

 

 

[Fuzebox]

For now i wont be upgrading my internet to 10gbit. 1gb is fine for now. And locally i don't have any other devices above 1Gbit anyways. I can put in 10gbit cards and get a router/switch for that later. What standard? rj45 connectors? Not sure what you mean.

Ofc if there is a board out there with 2x10gbit for a fair price i might go for it. Id be happy with 2x 1 or 2.5gbit too.

With a MiniITX only supporting one expansion card the board would have to have 2 integrated adapters. I might want to put in a SAS, another nic or something else.

 

Know what, im gonna sleep on this. Any suggestions are ofc very welcome.

[Electronics Wizardy]

3 minutes ago, Fuzebox said:

For now i wont be upgrading my internet to 10gbit. 1gb is fine for now. And locally i don't have any other devices above 1Gbit anyways. I can put in 10gbit cards and get a router/switch for that later. What standard? rj45 connectors? Not sure what you mean.

Ofc if there is a board out there with 2x10gbit for a fair price i might go for it. Id be happy with 2x 1 or 2.5gbit too.

With a MiniITX only supporting one expansion card the board would have to have 2 integrated adapters. I might want to put in a SAS, another nic or something else.

 

Know what, im gonna sleep on this. Any suggestions are ofc very welcome.

There is 10g base-t for 10gbe over twister, pair, and 10g base-lr and 10g base-sr which are over fiber. There incompatible. 

 

Id probably stick with 1gbe for now unless you have a lot of storage. When I do full backups of my ~40tb server, it takes a long time if I was limited to 1gbe. 

[Sophia_Borjia]

Ryzen non- pro apu don't officially support ecc.  Pro apu and ryzen cpu do support ecc, issue is more if motherboard supports it. 

An am4 server board and ryzen apu supports ecc fine officially, gaming motherboards may be spotty.  Motherboards are also spotty about posting headless, some will.