Looking for a 3-node cluster setup, custom build, please review my choices.

MG2R · January 16, 2022

Right, so, I'm trying to piece together a new cluster to replace my old server.

The system I'm replacing:

Dell R510
16 GB DDR3 1066
Intel Xeon E5620 (4 core, 8 thread, 11 years old, 2.4 GHz)
4 x 4TB + 4 x 2TB hard drives, running ZFS (3x4TB raidz1, 2 mirrors of 2x2TB, one 4TB hot spare)
Runs Docker Swarm with
- Nextcloud
- Plex (1 concurrent transcode, 1080p is the maximum)
  - + content... management system
- Gitea
- Factorio server
- UT99 server

As you can imagine, this system is pretty much at the very edge of its capabilities with that selection of software. I'm looking to replace it with a cluster of at least three machines so I can run HA Kubernetes with Ceph/Rook as distributed storage.

Per machine requirements:

Rack-mounted, smaller == better, as I may end up with co-location, which is priced per Uj
At least redundant 10 Gbps networking
16+ GiB RAM ECC
Room for at least 4 drives
500+ GiB NVMe drive, probably 1 TiB

One of the machines will be outfitted with a GPU, to run a gaming VM. Maybe I'll put in a second GPU for Plex, if needed. Looking into supporting at least 2 concurrent 4K transcodes.

If possible, I'd like to minimize power draw. Electricity in Belgium is quickly becoming stupidly expensive.

Currently I'm looking at this:

Arock Rack B550D4ID-2L2T
AMD Ryzen 5600G
2x 8GiB ECC DDR4-2133

Which would run me about 1300-1500 euro per machine before a GPU. What do you guys think about this? Are there better options for what I'm trying to achieve here?

Electronics Wizardy · January 16, 2022

Is there a reason why you want a cluster? A single node will be cheaper, use much less power, and use much less rack space. Thats what Id go with here.

Have you looked at the barebones asrock rack ryzen systems? I have one colocated now and its working great.

Id get the non g parts, the asrock rack boards have a gpu on board, and then you get ecc support with the non g parts.

But if I had to go with a cluster, Id get 3 of the asrock rack bare bones, get a 5600x/5900x in it,Id go with 32-64gb of ram as thats normally the first limiting factor. Then you can put 4x 3.5in hdds and 3x 2.5in ssds in those systems for storage, and m.2 drives for boot.

MG2R · January 16, 2022

I specifically want a cluster to play around with distributed storage, HA VMs, and so on. The idea is to learn on this while also use this to replace my current aging server.

I overlooked that I didn't need an APU, thanks for letting me know!

Interesting you mention those Asrock Rack barebones. Ever since getting the R510, I've made a point of not looking at pre-built servers or barebones because the non-standard server parts annoy me to hell. Especially the RAID card for the Dell has been problematic for me.

However, it seems the Asrock Rack barebones use pretty standard parts throuhgout. That makes them very interesting. It seems like it would be a touch more expensive, but the 1U case has drive sleds accessible at the front rather than internal hardmounts, which I was going to go for.

Interesting, I'll take it into consideration. Thank you!

Nick7 · January 18, 2022

16GB RAM for virtualization server is really bare minimum to run anything on it.

I'd suggest minimum of 32GB RAM, preferably 64GB+. Even if that means one less server.

MG2R · January 27, 2022

On 1/18/2022 at 7:51 PM, Nick7 said:

16GB RAM for virtualization server is really bare minimum to run anything on it.

I'd suggest minimum of 32GB RAM, preferably 64GB+. Even if that means one less server.

For Kubernetes/containers it really doesn't matter that much, though, as you're not running full VMs per use case. There will be the controlplane overhead, but using K3S, that should be minimal

I'm planning on replacing a single machine with 16 GiB of RAM with multiple of 16+ gig each, I won't be RAM-constrained, except for the system that going to run the virtualized gaming rig.