Accelerate AI Inference: Fast Model Loading on Amazon EKS

In the world of AI inference, the speed at which your models load can make or break your application’s performance. Long loading times can lead to increased latency, which is unacceptable in production environments where real-time processing is critical. This is where optimizing model loading on Amazon EKS becomes essential. By addressing the two primary bottlenecks—weights loading from S3 and the compilation of models using PyTorch's torch.compile—you can drastically improve your inference times.

When we instrumented the pod startup path on p5.48xlarge instances, we found that for a 64 GiB model, loading weights from S3 to GPU took approximately 29 seconds, accounting for 35% of the model startup time. In contrast, torch.compile consumed around 53 seconds, making up 65% of the startup time. For larger models, like a 203 GiB model, the situation worsens: weights loading balloons to about 423 seconds, which is 92% of the startup time, while torch.compile drops to just 34 seconds, only 8% of the time. This stark difference highlights the need for strategies that can mitigate these delays, such as using the Run:ai Model Streamer, which is already integrated into vLLM and SGLang.

In production, you need to be aware of these timings and plan your deployments accordingly. The integration of SOCI (Seekable OCI) can help accelerate container image pulls, but the real gains come from optimizing model loading. Be mindful of the instance types you choose and the size of your models, as these factors will directly impact your inference performance. Keep an eye on the versions of tools you are using, as updates may introduce new optimizations or features that can further enhance your setup.