Chat, batch, and agentic traffic can run on the same graphics processing unit (GPU) without the chatbot waiting for the batch job to complete. Built on llm-d flow control and generally available in Red Hat AI 3. 5, priority-aware serving decides which request goes first when the pool is full, and applications change nothing. On a NVIDIA H100, 95% of premium requests got their first token in under half a second, and batch rejections fell from 48,224 to 0. Sooner or later, every shared inference platform gets the same question from 2 teams at once: Whose request goes first?
A foundation model on 1 cluster now often serves the customer chatbot, the batch summarization pipeline, the agentic retrieval-augmented generation (RAG) workflow, and the internal copilot. They share a model, but they don't share performance requirements. A chatbot needs its initial token in under a second. A batch document pipeline cares about throughput per dollar, not any single request. An agentic workflow sits idle for minutes, then fires 5 concurrent chains. A shared pool with no notion of priority treats every request the same way: 1 queue, 1 dispatch order.
When a batch job floods that queue during a traffic spike, the chatbot's requests wait behind it. The rational answer has been a separate GPU pool per workload type, each sized for its own peak. It works, but it also means every pool carries its own reserve for traffic that is not there most of the time, and no pool can borrow another's idle capacity. Every new workload type adds another pool and another reserve. That headroom is what a shared pool can reclaim, as long as the pool knows whose request goes first. So how does the pool know whose request goes first? The application never has to tell it.
That call moves to llm-d flow control, a layer in front of the model servers, and the application keeps calling the same endpoint it always has. The operator gives each workload a priority in an InferenceObjective Kubernetes resource, sets the saturation threshold (the point at which the pool counts as full), and picks a fairness policy. The gateway reads a caller's credentials, and tags each request with a priority and tenant.
