What exactly do you wish to get input on? Reliability, cost effectiveness, availability? You also mention maintaining production hardened hardware?

Let's start somewhere with one of these.

Availability of services in cluster is reliant on many things:

• Load Balancing
• Rolling updates
• Replication
• Scaling
• etc

Reliability is ensuring efficient monitoring, good recovery and fallback mechanisms, self healing, etc. Realistically you need to make sure you have n amount of replicas of a service in multiple physical locations, all backed up.

Cost effectiveness involves optimizing resource utilization, scaling efficiently, and managing infrastructure in a way that maximizes value for the resources consumed.

My suggestion:

1. Automate creation of the cluster. If you want to learn industry standards, you can take a look at terraform or ansible and one of the public cloud providers. K8s was designed to run on a cloud provider ( even tho it can also be ran locally ofc ).
2. Deploy a CSI
3. Ingress
4. Certificates and ssl
5. Deploy a monitoring solution and make sure that it survives you unplugging one of the servers from the plug.
6. Repeat step 5 for other apps and stumble through the unique challenges that you'll face with scaling most of them

Side Note: take a look at k3s

It's not what you think. It can run a lot of things but it's so incredibly slow, especially if you don't have a ssd paired up, that for me it was unbearable.

Maybe one negative experience here:

I've used self hosted livesync for months before migrating to actual Obsidian Sync.

Reason being my vaults kept getting off sync and I had to keep rebuilding. Not to mention that moving files between always ended up being super tricky and would work half of the time. I tried really hard to like it but it just didn't stick.

The plugin sync was also horrible.

I wonder if you've also had any issues?