When you scale an AI inference workload on Kubernetes, every new pod must load model weights into GPU memory before serving its first request. For large language models, that means moving 60–200 GiB from Amazon Simple Storage Service (Amazon S3) to GPU memory. With default Run:ai Model Streamer settings, this takes 80–460 seconds on p5. We investigated where cold-start time actually goes, what causes the bottlenecks, and what configuration changes eliminate most of the wait.

Two changes, neither requiring code modifications, cut cold-start time for a 64 GiB model using Run:ai Model Streamer, a supported model loader in vLLM and SGLang. Initial launch dropped from 82 seconds to 65 seconds, and subsequent launches on the same node dropped from 82 seconds to 16 seconds. Results scale further with larger models, as shown in the following results tables. This post explains what we found and why we made the choices we did.

For the configuration itself (environment variables, concurrency calculations, YAML manifests, and instance-specific recommendations), see Accelerate model loading on Amazon EKS in the Amazon Elastic Kubernetes Service (Amazon EKS) User Guide. When you hear “200 GiB model”, the natural assumption is that downloading it is the bottleneck. Big file, network transfer, slow download. We started with the same assumption. The bottleneck in the stack actually varies by model size. We instrumented the full pod startup path on p5. 48xlarge and measured each phase independently.

Node launch (Karpenter) and container image pull are addressed by other efforts (Karpenter provisioning and SOCI (Seekable OCI) parallel pull ), and CUDA graph capture is a small fixed cost. That left two phases we could improve: weights loading from S3, and torch. compile (PyTorch’s built-in model compiler). For a 203 GiB model (Llama-4-Scout, TP=4 where TP is tensor parallelism, splitting the model across GPUs): The bottleneck flips. For models under ~100 GiB, torch. compile dominates startup. For larger models, weights loading dominates. This happens because torch.