Achieving 30-Second LLM Cold Starts on Kubernetes with Fluid

In the world of cloud-native applications, cold starts can lead to frustrating delays, particularly when dealing with large language models (LLMs). NetEase Games tackled this challenge head-on by implementing Fluid, a Cloud Native Computing Foundation (CNCF) incubating project designed to streamline dataset and runtime management in Kubernetes. By automating deployment and lifecycle management, Fluid enables rapid scaling and efficient resource utilization, making it a game-changer for performance-sensitive applications.

Fluid operates by automating runtime deployment and lifecycle management while supporting cache elasticity through mechanisms like Horizontal Pod Autoscaler (HPA) and Kubernetes Event-driven Autoscaling (KEDA). This allows for data-aware scheduling, aligning compute placement with cached data. Additionally, Fluid provides prefetch workflows that cater to scheduled, event-driven, and proactive warm-up strategies, optimizing model-loading patterns for frameworks like vLLM and SGLang. This targeted approach ensures that the necessary data is readily available, significantly reducing cold start times.

When deploying Fluid in production, be mindful of its operational capabilities compared to alternatives like Alluxio, which may lack the same level of control. Fluid’s focus on cache elasticity and data-aware scheduling is crucial for achieving those rapid cold starts. However, always evaluate your specific use case and performance requirements to ensure Fluid aligns with your operational goals.