When large open weight models first captured attention, the industry was faced with a challenge. Powerful open models were becoming available, but technologists lacked an open inference serving system that could run them efficiently. Having spent years building inference engines for CPUs, my team and I redirected all of our energy toward the vLLM project. Together with the community, vLLM capabilities expanded to become the de facto inference server.

Today the vLLM project, now part of the PyTorch foundation, has a vibrant community of companies and developers actively adding support for new models, hardware, and research techniques. As Red Hat continued this work, conversations with our enterprise users exposed a new reality. Getting a single server to perform efficiently was table stakes. The next challenge was orchestrating model serving at cluster scale. Last spring, we announced the llm-d project to address this, developing an open source, Kubernetes-native orchestration layer for distributed LLM serving across multi-tenant production clusters.

Today, Red Hat and our community partners announced the release of llm-d 1. Reaching this milestone reflects a project hitting full maturity, shaped by hardened code, stability proven under real production load, and widespread adoption across an expanding contributor community. Open source AI serving has crossed from an experimental science project into dependable infrastructure you can build on. In early open source experiments, teams measure progress through isolated benchmark runs on pristine hardware.

Keeping an AI service healthy on day 2, when hundreds of automated agents and human users contend for shared resources, requires an entirely different operational discipline. Platform teams came to us with a clear dilemma: What happens when a developer debugging code on an interactive session gets stalled because an autonomous background agent flooded the GPU pool with dozens of parallel tool calls? Building for day 2 means solving these friction points directly in the request path. Over the past year, our contributor community focused on intense code hardening.