AWS Cuts LLM Cold Starts From Minutes to Seconds

Amazon Web Services said Sept. 10, 2026 that SageMaker HyperPod now supports model caching, a feature meant to stop new inference pods from spending most of a scale-out event downloading model weights and container images before they can serve a single request. Kareem Syed-Mohammed, a principal product manager at AWS, and four co-authors wrote that cold starts on large models had been taking 25 to 30 minutes or longer, with models above 600 gigabytes needing more than 30 minutes just to download, before a pod could accept traffic.

The fix is two independent caches. A weights cache copies model weights onto the local NVMe drive attached to each cluster node, so a new pod reads from storage at roughly 7 gigabytes a second instead of pulling from Amazon S3 or FSx over the network. An image cache pre-pulls the container image onto target nodes via a DaemonSet, skipping the Amazon ECR download entirely. Both are optional and configurable through a modelCacheConfig block on the HyperPod Inference Operator, and both fall back to the original network path if a pod lands on a node without the cache warmed, which keeps a cache miss from becoming a failed deployment.

The numbers AWS is publishing

Across models ranging from 57 GB to 145 GB, AWS reported roughly a 60 percent faster scale-out with the weights cache enabled. The image cache, separately, cut pull time by up to 97 percent, removing more than two minutes per pod that would otherwise go to an ECR download. Local NVMe capacity on HyperPod instances runs from 250 GB up to 30,000 GB depending on instance type, which is the ceiling on how much of a fleet's model catalog can sit pre-warmed at once. Model caching is generally available now in every region where SageMaker HyperPod runs, with no separate preview stage to wait out.

Metric	Reported figure
Weights-cache scale-out speedup, 57-145 GB models	about 60 percent faster
Image-cache pull-time reduction	up to 97 percent, 2+ minutes saved
Cold start before caching, large models	25 to 30+ minutes
Local NVMe read speed vs. network download	about 7 GB/s

How this compares to the academic state of the art

AWS did not invent the underlying idea, and its own numbers are modest next to the research literature on the same problem. ServerlessLLM, a system built mostly by researchers at the University of Edinburgh and presented at the USENIX Symposium on Operating Systems Design and Implementation in July 2024, reported latency reductions of 10 to 200 times over existing serverless inference systems, using the same core idea of keeping checkpoints on local, near-GPU storage rather than fetching them from a remote store on every cold start.

The gap between a 60 percent improvement and a 200x one is mostly a gap in ambition, not physics. ServerlessLLM's multi-tier design spans GPU memory, host DRAM and local SSD simultaneously, with a scheduler that places new inference workers on nodes already holding a warm copy of the needed checkpoint. AWS's version, by contrast, is a single NVMe tier bolted onto an existing Kubernetes-based orchestrator, built to be safe and generally available across every HyperPod region on day one rather than to hit a research-paper ceiling. A production cloud service optimizes for the caching layer never being the reason a customer's deployment fails; a systems paper optimizes for the best number reviewers will accept.

The denominator is how often a fleet scales, not how fast one pod starts

The headline percentage in a cold-start announcement always describes a single event, but the number that actually decides whether this matters to a given customer's bill is how many scale-out events a fleet triggers in a day. A service with a stable, provisioned baseline and rare autoscaling barely notices a 25-minute cold start becoming instant, because it almost never pays that cost. A service that scales aggressively on bursty traffic, such as a chat product with a predictable daily peak, might trigger dozens of scale-out events, each one previously leaving a GPU idle and unbillable-to-the-customer for half an hour while weights downloaded. For that workload, model caching does not just make one event faster; it changes how many GPUs the fleet needs provisioned at the peak, because autoscaling becomes fast enough to actually lean on instead of over-provisioning around.

AWS's own post does not say what the caching layer costs to run, and NVMe capacity is not unlimited: a customer serving more distinct large models than a node's local storage can hold is back to deciding which ones stay warm and which fall back to a cold, 25-minute path. That tradeoff, not the topline 60 percent figure, is what determines whether this feature changes a real deployment's economics or just makes its worst-case demo faster.

What would change this read is AWS publishing a price for the cached storage itself, separate from the compute and NVMe it already bills, and a customer benchmark outside the 57-to-145-gigabyte band AWS chose to publish. Until either shows up, this is a real, generally available improvement with a plainly published ceiling, sitting well short of what the same idea has already been shown to do in a research setting three years earlier.

On the same theme of infrastructure built to survive its own scale, see how OpenAI's storage team rewrote a service in Rust to keep up with ChatGPT's request volume, and on the cost side of running inference at scale, how GitHub says it made Copilot's agent mode more cost-efficient without cutting task quality.