Project: Building mini kubernetes Cluster for home usage.

[Smurphy]

Hi folks,

as Hobby, for personal use and for deep diving into kubernetes and stuff, I intend to build a small cluster using x86 based SBC's for personal usage.

The Idea is to host on it Nextcloud, Web/SMTP/Imap/Matrix servers and more.
Currently I have these services running on a single device (6 years old) and I bet that sometime in the next 3 years - things will start to break apart (from experience - hosting my stuff for 25years now).
The reason I want to use that setup, is that a part from the master node/disk subsystem I could easily scale the setup when needed. And if a device breaks down, I can simply replace it. Also - I could start using a NAS in the LAN and a Server in the Service network. Of course, I'll have to rethink my current security procedures.


So - what I want to achieve is the following:
- Kubernetes cluster with 5 hosts SBC's.
- SBC's should have at least 4 cores, 8 GB ram (or 16 - but honestly, as I don't use any JAVA apps, I don t think I'll need more)
- Storage will be handled on the Master node or on the NAS Server (Don't know still). But I could take 2 2Tb SSD's on the master node
- Networking would be handled by a RB493G (Still have couple of these laying around here)
- PSU for these devices with 15V / 150W power supply. Anyone knows how to connect 2 of these together to make these redundant? One fails, the other takes over!
- I'll build the enclosure myself. Probably plexiglass or so (Easy to work).

I'll try to stay below 1000 Euro for the hardware.

Most affordable 4core SBC I have found so far is the ODROID-H2+ board (https://www.hardkernel.com/shop/odroid-h2plus/).
I tried to check for existing alternative using Ryzen CPU's - but the boards are designed for Enterprise business and usually start at 500 US for what I have in mind. So no go. If anyone could prove me wrong, I'll be glad!

I know it would probably possible doing that with ARM hardware - but even though I tend to build all my docker images myself, the software inside does not always run on ARM docker. Hence the choice to run on x86 hardware!

So - to my questions
- Anyone here has done something similar? Any pitfalls I should avoid (due to your experience with it)
- Which SBC's did you guys use
- Anyone knows how to connect to PSU together for HA?

any advice, hint etc. welcome.

[royce]

I can't really help with the hardware side of things, as I personally only deploy to cloud providers with managed k8s these days using stuff like terraform. Some tips:
* Definitely make sure your sbcs aren't ARM. Sourcing images in ARM is a problem, and you might need to build many yourself which may or may not be a problem. Those odroid look fine for the job though.
* Managing and deploying kubernetes is hard. Look into tooling to automate it, such as Canonical's Juju. Particularly, tooling that allows you to set up your infra as code. Manual, piece by piece set up of kubernetes is pretty hard.
* Managing storage in kubernetes is also hard. If you don't need ultimate performance, your NAS over a decent network will be more than enough. Master nodes are busy - don't use them for anything else. They don't need to be as beefy as your worker nodes though - it's better to have 2 smaller master nodes if you can. If your master node goes down, so do your services.
* Look into k3s, which is a slimmed down version of kubernetes. It might just be the thing you want.

[Smurphy]

yeah. Thx. In fact - I came across the ODROID-H2+ devices. Small and affordable J4115 (celeron, x86). I can put up to 32Gb ram onto it, has NVMe and 2 S-ATA connectors and 2x2.5Gbps ethernet connectors: https://www.hardkernel.com/shop/odroid-h2plus/
And the when looking up the hardware I'd need to get, I saw that there is https://www.picocluster.com/collecti...co-3-odroid-h2 doing some micro-clusters setup for that device.
They have a 3 device on the shops page, but after contacting them, they will bring out a 6 device setup next week (well, hopefully it work work out).
If that works - I'll go for it.

Question will be what to use for persistent storage. I could use my NAS for that (even though the disks are only HDD). But I could also hook up 2 SSD's onto my NAS and connect these through E-SATA (HP Microserver Gen 8 with a Xeon CPU). I'd eventually have to add a dedicated/faster network card to the Hp Microserver to take advantage of the speed.

Regarding the setup and stuff. As my servers & NAS run fine so far, I will primarily use that cluster as test-bed to check out what the best setup/toolset will be.
I'll put the details on my build blog as soon as some hardware is there :}

[royce]

If you want to play around with setting up the actual apps you want on your cluster prior to actually purchasing any hardware, you could do far worse than microk8s. I model a lot of deployments using it, and it can cluster as well to simulate many nodes.

[Smurphy]

Thx. will check it out once I have a base to play with

[Smurphy]

Question to those in the forum. Anyone knows where I could get a couple of SAPPHIRE BP-FP5 boards?
Seems these are unavailable, and if there are one or 2 samples available, prices go up to 350 euros a piece.