Recommendations for scalable storage, multi-node and load balancing on k8s?

[aDataWat]

Hey everyone, was hoping for some help as I try and scale a small business using #kubernetes

 

We provide a service where each client is allocated their own instance of a database on Kubernetes. Currently, we utilize hostPath storage by mounting a local directory on the node to store data. However, we've realized that this setup may not scale well as we plan to add more nodes to our cluster.

 

We've explored High Availability and Mayastor for on-node distributed storage, but these require a minimum of three nodes. While this could be a viable path, it necessitates a load balancer in front of the nodes which is an additional consideration.

 

We do currently plan on going from 1 server to 2, so we recognise that HA won't be available for us right now

 

We are now actively seeking advice on:

 

1.Alternative storage solutions that would facilitate smooth scaling as we add a second node, and potentially more nodes in the future.

 

2. Effective strategies for load balancing in a multi-node setup.

 

3. Any experiences or insights on transitioning from a single-node to a multi-node setup, particularly focusing on data storage and management.

 

Your suggestions and recommendations on potential solutions or improvements to our current setup would be amazing

[leadeater]

If you only need storage load balancing then Gluster and Ceph would likely be good choices.

 

6 hours ago, aDataWat said:

While this could be a viable path, it necessitates a load balancer in front of the nodes which is an additional consideration.

Is that for the storage solution you were looking at or to load balance container access? I assume the second.

[Jarsky]

I'd recommend having a look at Longhorn https://www.rancher.com/products/longhorn

Check out TechnoTim's stuff 

 

 

 

[aDataWat]

18 hours ago, leadeater said:

If you only need storage load balancing then Gluster and Ceph would likely be good choices.

 

Is that for the storage solution you were looking at or to load balance container access? I assume the second.

So it would be that the nodes are in different locations with 2 servers having 2 different IPs so it's how do we know where to send each request and having a bit of disaster recovery as if one machine goes down for some reason it would spin up the required containers and re-route the traffic with 0/minimal downtime or interruption.

We're doing this self hosted, I've updated the initial post as I realise this is an important thing to note

[leadeater]

3 hours ago, aDataWat said:

So it would be that the nodes are in different locations with 2 servers having 2 different IPs so it's how do we know where to send each request and having a bit of disaster recovery as if one machine goes down for some reason it would spin up the required containers and re-route the traffic with 0/minimal downtime or interruption.

We're doing this self hosted, I've updated the initial post as I realise this is an important thing to note

If they are going to be distant servers then I think @Jarsky suggestion is going to be suitable, Gluster and Ceph both can do remote replication but they have minimum scale sizes for that which would realistically be 3+3 (6 total) servers, at least for Ceph.

 

You wouldn't want anything that is synchronous replication unless the distance and latency is very low or you'll destroy your storage performance.