Join our product leaders to architect your agentic future Today at the PyTorch Conference, we released llm-d 1. 0, an open-source, Kubernetes-native stack for distributed LLM inference. Co-founded by Google with Red Hat, IBM and others, we worked with a community of more than 500 contributors over 25 organizations on this Cloud Native Computing Foundation (CNCF) Sandbox project, bringing together two important communities: PyTorch, where vLLM and SGLang serve models, and Kubernetes, where distributed systems run and scale in production. 0 arrives at a time when agents are changing the nature of inference.
A year ago, most inference traffic was chat: one prompt in, one response out. Today, workloads are shifting to multi-step agents and reinforcement learning (RL) rollouts that fan out across tools, share long prefixes, and arrive in bursts. Serving them on a single machine — or behind a generic round-robin load balancer — breaks down fast. Replicas recompute the same prompts, accelerator memory fills up with duplicate state, and one bursty agent loop can stall interactive users. llm-d solves this for teams serving their own models by turning a cluster of model servers into a coordinated serving pool.
During the PyTorch Conference keynote, we showed this in action with 1,000 sandboxed agents on Agent Substrate on GKE hitting a single llm-d endpoint across vLLM, SGLang, and a mixed fleet of Google Cloud TPUs and GPUs. As traffic scaled across the three stages below, cache-aware routing and tiered KV cache kept each agent's steps on the replicas holding its context, while flow control queued bursty agent loops so interactive requests stayed fast. Figure 1: Evolution of distributed inference from round-robin load balancing to cache-aware routing, tiered KV cache, and priority flow control in llm-d 1.
0 llm-d already delivered prefix- and load-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism and batch inference across Google Cloud TPUs and GPUs. These capabilities provide: The components are designed so that you can compose them into your own bespoke production environment. Keep your engine, model and cluster, and adopt only the serving components and configurations you need. Across the areas below, Google engineers worked with the llm-d community to connect upstream Kubernetes primitives, PyTorch serving engines, a
