Distributed AI training workloads impose complex scheduling requirements that Kubernetes's built-in scheduler can't meet. Kubernetes schedules pods individually and independently, but distributed training introduces two requirements that break this model: Pods must land on hardware with the right inter-GPU bandwidth, and all pods must be scheduled simultaneously. If either requirement goes unmet, training stalls or runs far below the hardware's potential.

Topology-aware scheduling (TAS) and gang scheduling address these two requirements and help organizations extract full value from GPU capacity, which represents a growing share of infrastructure spend. TAS shapes the relative placement of pods to enable fast communication between GPUs. Gang scheduling helps ensure that all pods in a distributed job start running simultaneously, preventing the partial starts that leave GPUs allocated but idle. Together, they give Kubernetes the placement precision and coordination that distributed workloads require.

In this post, we'll look at the scheduling problems that AI training presents for Kubernetes and how TAS and gang scheduling, implemented by Kueue and the Coscheduling plugin , solve them. We'll explore: We'll also cover the evolving capabilities of Kubernetes native scheduler to support AI training workloads. In most batch workloads, pods operate independently. A pod can fail, be rescheduled, or be replaced with no effect on the rest of the workload. The Kubernetes scheduler handles this model well by placing pods one at a time on any eligible node without coordinating across pods.

But the pods in an AI training workload are interdependent. They need to be placed on hardware with appropriate inter-GPU bandwidth, and they all need to start simultaneously. The following sections explain these requirements and how Kueue and Coscheduling address them by augmenting Kubernetes' scheduling capabilities. The physical interconnections between nodes are a key factor in the performance of AI training workloads.