proxmox ve Overcoming Hardware Challenges for OKD Cluster Setup

[Nevah5]

Introduction
Hey everyone!

So I am fairly new to self hosting stuff at home, I am talking about 2-3 years here. I have bought my first Raspberry Pi about 3 years from now and my small home server 1.5 years ago. I first ran TrueNAS Scale as the OS, but switched to Proxmox VE, because I had issues with accessing other pods hosted on the same issue. Talked about this issue with one of my teachers back then and he advised me to install Proxmox instead of trying to fiddle around with IP tables and stuff. Since then I am a huge fan of PVE!

Alright, so I then also started hosting other VMs on there, because I needed the unused computing power that I used for the pods on TrueNas Scale again. I started migrating some of my services onto an Ubtuntu-Server Docker instance. It's been like that since that point. Recently I started experiencing OpenShift at work and CI/CD pipelines with Tekton and wanted to also do something like that privately to strengthen my knowledge about k8s. I started researching and found the underlying project of OpenShift; OKD. Long story short; I checked the minimum requirements for OKD and realized that I am short of a few GB of RAM...

 

My Goal

I would like you to quickly take a look at the table 1 and 2 on the following official documentation: https://docs.okd.io/4.14/installing/installing_bare_metal/installing-bare-metal.html#installation-machine-requirements_installing-bare-metal

For my cluster, I would need 3 control plane nodes and at least 2 workers. That would be a total of 14 CPU-Cores and 64 GB of RAM. I also have other VMs running on my system, that I have listed below this section. So there needs to be slightly better hardware, just to be safe.

 

Current Specs
Case: Fractal Define 7 XL Black
CPU: AMD Ryzen 5 5600 G, AM4, 3.9 GHz, 6-Core
RAM: Corsair Vengeance LPX, 4x 16GB, 3200 MHz, DDR4-RAM, DIMM
Motherboard: ASUS ROG STRIX B550-A Gaming, AM4, AMD B550, ATX
PSU: be quiet! Pure Power 11 CM, 600W

Storage:
- 2x Intenso top performance 512 GB, M.2 2280
- 2x Samsung 870 QVO, 1000 GB, 2.5''

- 6x Seagate IronWolf 8TB, 3.5'', CMR

Other:

- 2x Delock Host Bus Adapter 2 Port SATA PCIe
- 3x Delock Molex to SATA power cable

Virtual Machines

NAS

CPU: 1 Core [host]
RAM: 8GiB [balloon=0]
Storage:
- 6x HDD pass-through
- local-zfs (onto m.2) 32G virt. disk

 

Docker Services (Ubtuntu Server)
CPU: 3 Cores [x86-64-v2-AES]
RAM: 14GiB
Storage: local-zfs (onto m.2) 128G virt. disk

 

Adguard (Ubtuntu Server)
CPU: 1 Core [x86-64-v2-AES]

RAM: 2GiB

Storage: local-zfs (onto m.2) 32G virt. disk

 

(The SSDs are currently unused, because I wanted to put the cluster onto those. Distributing the control planes and worker nodes onto them, having no ZFS pool for possibly a better performance?)

Issues

1. My system was experiencing a high I/O rate, peaking at around 15-20% when doing high intensive writes. I had some MC servers hosted on there at some point, with large worlds and when they saved, all VMs were lagging to their death (almost). This was the case when I had the SSD1 and SSD2 as a RAID1 and I read that this can cause huge lag (possibly from ChatGPT, I cannot remember). Well so yesterday I migrated away from RAID1, also because I need more storage for my OKD cluster.

2. Another issue is that I had to buy those PCIe SATA adapters, because I didn't know that 2 of my 6 SATA motherboard connectors are going to get disabled, when using 2x m.2 slots. (I decided later that I am going to buy SSDs for my system).

3. Yesterday, a friend of mine pointed out, that my CPU only has 2 lanes and that this is probably going to be a bottleneck for my RAM. He advised me to do some more research about this topic though and as you can see I also don't know that much about hardware.

4. About the RAM issue, I would actually buy 2x 32 GB sticks here, and replace 2 of my 4 16GB sticks with them.

5. If I understood Proxmox VE correctly, it emulates certain CPU types, so the operating system would think that it has 4 cores, if I for example give it 1 socket with 4 x86-64-v2-AES cores. Also I still don't quite understand that part yet, with what CPU type to choose. Maybe I can get some advice here as well.

6. I actually want to give my NAS VM way more RAM, for faster caching, but it's not necessary really.

Conclusion

I would be very happy if I could get some advise about my current situation and maybe if you could address some other concerns that I maybe haven't stumbled into yet. I am willing to spend around 500$ for my upgrades (or more maybe, but I'll have to save up some more money then).

I've also thought of maybe moving the NAS onto another Proxmox system (including the VM with the Docker services), but this will be more expensive. But this makes the system available for the cluster (with a small RAM upgrade).

[unijab]

Ive been meaning to try it myself soon, but i noticed this the other day...

 

It does say 16GB of RAM tho.

 

https://fedoramagazine.org/okd-on-fedora-workstation-with-crc/

 

------

 

In 2022, I was able to create a single node OKD with 7 VMs, but to get it to reliably bootstrap, I needed 128GB RAM on the host to divide among the VMs.

 

And to be honest, setting up the supporting stuff before okd was more involved than deploying okd.

 

Load balancer, DNS, DHCP, internal registry (if your internet isnt fast enough to prevent bootstrap from timing out)... etc

 

My setup was 5900x , 128GB RAM, 1x 1TB nvme, 4x 2TB nvme

 

[Nevah5]

14 hours ago, unijab said:

And to be honest, setting up the supporting stuff before okd was more involved than deploying okd.

I still want to try it at least, but still don't know how I should conquer my lack of computing power.

[unijab]

Either way you go... (single host or multiple hosts) having a spare machine to run the support infrastructure would be helpful.

 

...

 

For a full on cluster: If getting multiple computers (ebay, optiplex, nucs, etc) is possible.. that might be the simple way.

 

I guess that depends on your budget...

 

...

 

But if you want to try CRC

 

it has way lower standards for hardware

 

[Nevah5]

Just now, unijab said:

But if you want to try CRC

Oh~ I didn't know this was a thing. Then I will certainly try CRC first, before going all in. Thanks!

[unijab]

yeah crc was the link I posted above

[Nevah5]

On 4/3/2024 at 1:01 AM, unijab said:

yeah crc was the link I posted above

Yeah now I got what you meant. I will try the CRC for now (will probably install it this weekend), but I still consider to setup a whole cluster. The CRC is only for development purposes and I am afraid that it isn't very stable and redundant.

[unijab]

7 hours ago, Nevah5 said:

and redundant.

 

Neither would be a single system with 7+ VMs.

But its about learning