Rebuilding the homelab
About 18 months ago, my OpenStack install completely fried itself after an update, and I really had no desire to try and fix whatever had been destroyed. The whole thing had been more trouble than it was worth, and I wasn’t really using it.
The server remained dormant until a few weeks ago, when I decided to reinstall Ubuntu and bring up a kubernetes cluster.
I’ve spun up a number of services on this new cluster, both my wife and I are
actively using them. So I count that as a pretty big win. I’ve also managed
to do away with all my hacked together Ansible provisioning scripts, with
Traefik nicely handling all ingress to the services now rather than a bodge
job with iptables that would evaporate every time OpenStack would restart
its network layer.
I think, even though the initial set up was frustrating and took me a while longer than I anticipated, it’s definitely simplified a large amount of my home projects. I commit to Forgejo, Argo picks up the changes and does a deploy. External DNS and Cert manager handle the fiddly bits around hostnames, and I have Longhorn backing up my persistent volumes to my NAS. It’s all running smoothly (touch wood) and is really easy to harden with network policies and security contexts. Not to mention my new Authentik install providing me with easy SSO across all my apps.
The other thing I found interesting was how easy it was to add a new node to the cluster. K3s generates a token, you just run the starter script with the endpoint and token and it joins. Aside from a slightly frustrating experience with Cilium taking over every single NIC in the machine and locking me out, it was made simple.
I’ve learned what I set out to learn with this little project, hopefully I’ll be able to learn a little more about hardening services and how to better deploy services into the cluster.
The learnings
I think the most important thing I’ve learned is that AI models can’t do kubernetes templates very well. It’s meant that I’ve learned how to write them by hand, and have even configured a few of my own nvim snippets to get a new resource moving quickly.
I’ve also taught myself about CRDs, Helm charts, kustomize overlays and general K8s networking. I’ve learned A LOT in the last couple of weeks just fighting the cluster. It has rewarded me for my persistence, things make a lot of sense in my IaaC repos, and I have a system I can bring back up very quickly if I need to tear it down. I think my next step is to dive into the Cilium source code and learn more about eBPF, seems like my jam.
The Blog
With this new setup, I decided to tear down my old S3/Cloudfront set up. Ostensibly, my reasoning was to stop Jeff Bezos getting more of my money, but when it came down brass tacks about $1.50 of my $1.60 bill was Route53 hosted Zones, which I’m not going to get rid of. The container you’re reading this from is a tiny thing I think running on my rack server at the moment.
The other bastard I’ve removed from my blogging process is Gatsby, I’ve moved this over to Hugo and it was largely painless. Just had to modify a little bit of frontmatter and we were ready to rock.
While Gatsby has served me well, the automated build jobs I had running would periodically break and take significant effort to fix. While not a huge deal, after many years of having to debug and fix this kind of thing I was just sick of dealing with it. My heart goes out to Javascript developers, this just seems like a normal Tuesday for you guys.