AWS Cuts SageMaker HyperPod Cold Starts by 60% With Local NVMe Caching

Amazon Web Services made model caching for SageMaker HyperPod generally available on Sept. 11, a feature the company says cuts scale-out time by about 60 percent for models between 57 gigabytes and 145 gigabytes. A separate image cache cuts container pull time by 97 percent, saving more than two minutes on each cold start.

The feature works through two independent mechanisms, according to AWS's announcement: a weights cache that stores model weights on local NVMe storage for fast local reads, and an image cache that pre-pulls container images so a new instance does not have to wait on an Amazon ECR download before it can start serving traffic. Both fall back to their original sources automatically if the warm cache is unavailable. It ships through the HyperPod Inference Operator, enabled by adding a modelCacheConfig section to an InferenceEndpointConfig or JumpStartModel resource, and is available in every region where HyperPod runs.

Why Cold Starts Cost More on a GPU Than a CPU

A server that scales a stateless web service can pull a small container image and be answering requests in seconds. An inference server carrying a 100-gigabyte-plus model has to move that weight file from object storage to GPU memory before it can answer anything, and AWS's own numbers show that download dominating the startup budget: a model at the upper end of the 57-to-145-gigabyte range that AWS tested was taking, in the company's words, tens of minutes to start serving traffic before this release.

That delay matters specifically because of how LLM-serving autoscaling breaks down. A GPU can report itself as 60 percent busy by a standard load metric while its request queue is already backed up, because token generation, not raw utilization, is what determines whether a user is waiting. When traffic spikes and a scheduler needs a new instance to absorb it, every minute spent re-downloading a model the cluster has already loaded somewhere else is a minute of degraded response time with no corresponding safety valve.

The Number That Matters Is the Denominator

A 60 percent reduction sounds uniform, but AWS ties the benefit explicitly to model size: the announcement states gains increase as models get larger, meaning a small model with a fast cold start already sees less benefit than a 145-gigabyte one. That is worth stating plainly because vendor percentages without a baseline are close to meaningless; a 60 percent cut on a model that took two minutes to load saves 72 seconds, while the same cut on one that took 40 minutes saves 24. AWS did not publish the absolute before-and-after times for either end of its own range, only the percentage.

Cache type	Reported improvement	What it applies to
Weights cache	~60% faster scale-out	Models 57GB-145GB
Image cache	97% faster image pull	Container images (2+ min. saved)

The economics behind the feature are not abstract. Market data compiled by Thunder Compute put the AI GPU rental market at $52.04 billion in 2026, up from $34.62 billion in 2025, and found that dedicated GPU clouds charge roughly half of what hyperscalers do for the same H100 hardware, a gap the firm attributed partly to how efficiently each operator keeps accelerators busy. A GPU sitting idle during a model download is a GPU a customer is paying for and not using, on either side of that price gap, and it is the same idle-capacity problem behind Oracle's decision to triple its GPU deliveries to 300,000 units rather than let a growing backlog sit unserved.

Academic work on the same problem backs up the shape of the fix rather than AWS's specific figures: a paper on cold-start latency in vLLM identifies weight streaming, memory snapshotting, and pre-warmed local caches as the categories of technique that address the gap, which is the same taxonomy HyperPod's two-cache design falls into. Neither AWS's announcement nor the paper puts a dollar figure on what a faster cold start is worth per GPU-hour, which is the number that would let a customer compare this against simply overprovisioning warm capacity instead.

AWS did not say what the feature costs beyond the underlying NVMe storage and EC2 instances already billed, nor did it name a customer using it in production. Pricing and a named reference deployment are the two things worth checking before the next HyperPod update, the same gap GitHub left open when it cut Copilot's inference token costs without saying which infrastructure change bought the savings.