Posted on October 23, 2026 by By Anish Bista | CNCF Golden Kubestronaut GPU teams usually face a bad trade-off. Give every team its own Kubernetes cluster and you pay for a control plane per team. Put everyone in one shared cluster and you get weak isolation. Either way, Kubernetes hands out GPUs as whole cards, so a pod that needs 4 GB of an 80 GB GPU still blocks the entire card. This post shows a way out by combining two open-source projects: I built the whole thing on a single machine with no physical GPU, using HAMi’s mock device plugin to simulate the cards.

The result: two fully isolated tenant clusters, each scheduling fractional-GPU workloads through HAMi, with all eight tests passing. Everything below is reproducible: the exact versions, the commands that worked, the errors I hit and how I fixed them. One kind cluster hosts Kamaji and both tenant control planes; each tenant’s worker is a separate container that joins over the kind network and runs HAMi. Tenants never share a node or an API server. The only shared pieces are the management cluster and the kamaji-etcd datastore, where each tenant’s data is kept separate.

Kamaji solves the cluster problem and HAMi solves the GPU problem; they work at different layers, so they don’t overlap. The interesting part is where they meet. A tenant’s API server lives in the management cluster, but HAMi’s webhook lives on the tenant’s worker nodes. Every GPU pod creation therefore depends on the API server reaching back into the tenant through Kamaji’s Konnectivity tunnel. Proving that path works was the main goal of this lab. Everything runs as Docker containers on one Linux host; 4 CPU cores, 8 GB of free RAM and 20 GB of disk are enough.

HAMi’s real device plugin talks to the NVIDIA driver, so it can’t start without hardware. The mock device plugin skips hardware discovery. You describe fake cards in a node annotation, and it registers gpumem and gpucores capacity from that. HAMi’s webhook, scheduler and quota logic then run exactly as they would on real nodes. Two simulated setups make the tenants easy to tell apart: The build has five steps; step 3 creates both tenants at once, and steps 4 and 5 are repeated per tenant. Plain kubectl talks to the management cluster; --kubeconfig tenant-a. kubeconfig talks to a tenant.