Engineers at Amazon Web Services reduced container image pull times for machine learning workloads from several minutes to under a minute by parallelizing the download and unpacking stages in the image retrieval pipeline, according to a technical report published by The New Stack. The team tackled a production bottleneck on Amazon EKS where pods needed to be ready within two minutes, but fetching a roughly 30 GB container image alone consumed several minutes—time during which provisioned GPU accelerators sat idle while waiting for images to arrive. The improvements are now enabled by default on EKS Auto Mode for GPU and accelerated instances, and AWS contributed the core changes upstream to containerd and the SOCI snapshotter.
Modern ML inference images typically reach 20 to 30 GB in compressed size, with some climbing even higher, carrying a deep-learning framework, the CUDA stack, and sometimes model weights. Individual layers within these images often exceed 9 GB, while the GPU software stack alone—PyTorch, cuDNN, and CUDA—imposes a compressed floor of roughly 3 to 4 GB that no build optimization can eliminate. On accelerated instances with 100 to 400 Gbps of network bandwidth, pulling these images consumed several minutes before the application could serve its first request. The team profiled the image pull path and found that neither the network nor the registry was the constraint; instead, the real bottleneck was how the software used the hardware already available.
The report explains that containerd, the industry-standard container runtime, traditionally performed most pull operations sequentially: each layer moved through six stages—fetch, verify compressed bytes, write to disk, decompress, verify decompressed content, and extract files—one at a time. By default, containerd downloads up to three layers in parallel, but each layer uses a single connection, and unpacking remains strictly sequential across layers. The authors write that "because layers are not equally sized, the single largest layer becomes the long pole in the pipeline"—a 10 GB layer taking a minute to decompress on one core held up the entire image, even if other layers finished in seconds. The result was that at any moment during a pull, the node was bottlenecked on only one resource while others sat idle waiting their turn.
AWS's solution split large layers into fixed-size chunks fetched concurrently over separate HTTP range requests, writing each chunk directly to local disk the moment it arrived to keep the runtime's memory footprint constant regardless of image size. Once download finished in seconds rather than minutes, the team built an unpack path that decompresses and extracts all layers concurrently instead of sequentially, reducing total unpack time from the sum of all layers to roughly the time of the single largest one. This concurrent approach works because the overlay snapshotter—the default on EKS and most Kubernetes clusters—keeps each layer in its own separate directory, so unpacking one layer doesn't depend on another finishing first. With chunked parallel download, a large layer that previously took over a minute on a single connection finishes in single-digit seconds, and on larger images with faster storage, the gains are more pronounced because the gap between available hardware capacity and what a single connection can use is wider.
The report identifies two remaining serial bottlenecks: decompression of a single large layer, which means a lot of compute sits idle on a 64 vCPU machine when inflating an 18 GB layer, and integrity verification, which today requires a single sequential read over the entire compressed blob after all bytes have landed. AWS suggests that libraries like rapidgzip could locate block boundaries and inflate blocks across cores in parallel, while a tree-structured hash like BLAKE3 would allow computing the layer digest from independently hashed chunks so verification could run in parallel with download rather than as a separate pass. The authors note they've been contributing these changes upstream to containerd 2.2 because "this is where they belong: in the runtime itself, available to everyone by default rather than locked behind additional software." Organizations outside AWS can leverage the optimizations either through native containerd 2.2—which includes parallel download and unpack built into the runtime—or via the SOCI snapshotter on older nodes, with specific configuration examples provided for both AL2023 and Bottlerocket AMIs. The faster the community can collectively close the remaining serial stages, the sooner multi-gigabyte images stop being a deployment bottleneck for the entire ecosystem. Workloads that densely access framework code and model weights at startup will likely drive further runtime evolution, particularly as training and inference patterns continue fragmenting across organizational boundaries. Platforms that can't deliver sub-minute cold starts for these images risk ceding ground to architectures that treat the pull itself as infrastructure rather than application responsibility.