AI infrastructure conversations often start with GPUs. Accelerators provide much of the compute behind model training and inference, so the focus is understandable. But a production AI workload rarely starts and ends on a GPU. Data needs to be prepared and moved. Applications and orchestration services need to run. Models need to be loaded and served. Results may require additional processing. Platform teams are therefore not simply managing GPU workloads. They are managing heterogeneous workloads that depend on CPU, GPU, memory, storage, and networking working together.
For Kubernetes platform teams, the challenge is not just providing accelerators. It is matching the right resources to each stage of the workload. Data → CPU preprocessing → GPU inference → CPU post-processing → application The GPU may perform the most compute-intensive step, but overall performance depends on the complete path. If preprocessing cannot supply data quickly enough, the accelerator waits. If storage cannot deliver model artifacts efficiently, startup slows. If CPU, memory, or network capacity becomes constrained, adding more GPU capacity may do little to improve throughput.
Instead of asking: How many GPUs does this workload need? , platform teams should ask: What resources does each stage need, and where are the dependencies between them? That shift helps teams optimize the workload as a system rather than optimizing one expensive component in isolation. Different stages of an AI workload have different infrastructure requirements. CPU resources can handle data preparation, tokenization, retrieval, orchestration, application logic, and post-processing. GPUs and other accelerators are suited to highly parallel operations such as model training and inference.
Memory, storage, and networking determine how efficiently data and model artifacts move between these stages. Even inference itself is not necessarily one uniform workload. For large language models, prompt processing and token generation can have different compute and memory requirements. This creates an opportunity for platform teams to match resources to the work rather than forcing an entire AI pipeline onto a single infrastructure profile. Kubernetes provides a common orchestration layer for doing this.