Adding an inference replica creates capacity only after the new instance can serve requests. Between starting a process and generating a response, an LLM server may need to load weights, initialize GPU resources, compile kernels, capture execution graphs, and warm up the engine. Restarts, model switches, and scale-out can repeat this work. The community is addressing these costs at several levels. vLLM and SGLang are developing ways to reuse weights and initialized state. Serving platforms are building workflows around warm instances and snapshots.
Kubernetes, container runtimes, and the Open Container Initiative (OCI) are working on related interfaces and artifact questions. These efforts have different boundaries and are progressing independently. Understanding what each approach preserves helps explain both its benefits and its current scope. A typical startup includes several kinds of work. The server launches processes and imports libraries. It reads model files, constructs tensors, and performs any required weight transformations.
It initializes the execution runtime and communication between workers, allocates memory, and prepares the engine to serve. Some work happens specifically to improve subsequent inference performance. Compilation produces executable kernels; autotuning selects implementations for a configuration. CUDA Graph capture records GPU operations for later replay with lower launch overhead. The vLLM optimization documentation describes the tradeoff between startup work and steady-state execution performance. The balance depends on the model, software, hardware, and storage.
Once model files are cached locally, engine initialization may become more visible. Once weights can be reused in GPU memory, imports, tuning, and graph capture may account for more of the remaining delay. Startup optimizations can be grouped by the work they preserve: These approaches can appear together in a system, with integration conditions specific to each implementation. For example, an engine might reuse compiled artifacts while loading weights through a specialized loader. A snapshot workflow might discard an empty key/value (KV) cache before capture and recreate it after restoration.
