Eleven minutes, zero humans: Building a self-healing Kubernetes upgrade pipeline on Kairos

Posted on August 14, 2026 by Olivier Calzi | CNCF Golden Kubestronaut Once upon a time, upgrading a Kubernetes control plane meant staying awake for it. Watch etcd health the whole time, hoping quorum holds through every reboot. This week I found out whether the pipeline I built actually ends that story. It was bootstrapped with OpenTofu with three control plane nodes, K3s HA, Cilium CNI, all provisioned as code before a single workload ever ran. I think of it as the root of a platform that’s going to keep growing: more tooling, more workloads, more clusters hanging off it over time.

Roots need to be solid before you build on top of them. And in 2026, with the pace of CVEs landing across the stack, the thing I wanted most from this root wasn’t more features. It was a genuinely simple, genuinely reliable upgrade process. Something I could trust to run without me needing to relearn the steps every time a patch shipped. I wanted to use the most possible tooling from Cloud Native Computing Foundation (CNCF) to avoid any vendor lock-in and build upon my Golden Kubestronaut knowledge.

Three control plane nodes, k3s HA, running Kairos Hadron an immutable Linux distribution built around A/B partition upgrades and cosign-signed images. Kairos doesn’t patch in place. It writes a new OS image to an inactive partition and reboots into it. Rollback is just booting the old partition again. That’s a strong supply-chain story. But it only matters if the upgrade actually happens. My previous version of this pipeline had a subtle but dangerous bug: concurrency: 0 in the upgrade spec, which I assumed meant “one node at a time. ” It means all nodes simultaneously.

Three control plane nodes rebooted at once during a homelab test. That fix is the backbone of everything below. Gitea — self-hosted git, running on its own Kairos + k3s cluster, with a Gitea Actions runner executing CI. This is where every manifest, policy, and upgrade spec lives. io/kairos/hadron on Quay for new tags. When upstream publishes a release, Renovate opens a PR bumping two lines: the image tag and the metadata. Kyverno — a ClusterPolicy admission gate that rejects any upgrade CR whose image doesn’t match quay.