Posted on September 9, 2026 by Bingi Narasimha Karthik (Golden Kubestronaut, Adobe) and Ramkumar Nagaraj (Golden Kubestronaut, Adobe) It was a routine cost review. The slide showed the month’s GPU spend, the biggest line on the whole infrastructure bill, and someone asked a five-word question: “Are we using these things? The most expensive hardware we owned was also the hardest to see. The frustrating part is that the answer already existed.
Every GPU’s utilization had been recorded, every second, for months, all of it flowing into one central, infrastructure-owned Prometheus that held every metric for every team across thousands of namespaces. The data was right there. It just sat somewhere no tenant was allowed to look, because a store that sees everyone’s metrics can’t safely be opened to any one of them. So visibility split in two: When we finally went looking, we found a GPU that had sat at zero percent utilization for eleven straight days: allocated, powered on, doing nothing, and invisible to the team that owned it.
You can’t fix what you’re not allowed to see. Multiply that one idle card across a fleet, all of it drawing power behind green health checks, and “we’re not sure” becomes real money every month. This post is how we closed that gap: how we gave every team a safe, self-service view into their own metrics without handing them the keys to everyone else’s. No new metrics stack, no vendor platform, just CNCF-native pieces arranged so the people spending the GPU budget can finally see it. The obvious fix is to give every team read access to the central Prometheus.
We ran into two walls, each built from an individually correct decision. A Prometheus query endpoint isn’t namespace-aware: if a tenant can run one PromQL query, they can run any query, including one that reads another tenant’s request rates or capacity plans. “Everyone can read everything” isn’t a posture you can defend across thousands of namespaces. The second is scale, and it has a name every platform engineer knows: the noisy neighbor. The central Prometheus is already scraping and storing series for the whole fleet.