Unifying AI Training and Inference on Kubernetes: Lessons from China Merchants Bank

In the world of AI, managing resources efficiently is critical. China Merchants Bank has tackled this challenge head-on by creating a unified control plane that integrates Kubernetes with Kueue, KEDA, Prometheus, HAMi, and Fluid. This setup allows them to streamline model training, fine-tuning, and online inference, maximizing the use of their accelerator resources while minimizing idle time.

The architecture is built around several key components. Kueue handles training admission, queues, and quotas, ensuring that jobs don’t reserve capacity before they can utilize it. KEDA and Prometheus work together to scale online inference based on live demand signals, dynamically adjusting resources as needed. HAMi allocates shared accelerator capacity in fine-grained units, optimizing usage across different tasks. Fluid accelerates access to datasets, model weights, and checkpoints, reducing the time accelerators spend waiting for data. This combination allows for nearly 10,000 heterogeneous accelerator cards to be utilized effectively, significantly enhancing training density and resource efficiency.

In production, the integration of these tools can lead to substantial resource savings and improved performance. For instance, the Twinkle training framework allows multiple tenants to share a base model instance, cutting accelerator resource usage by 80% while increasing training density fivefold. However, engineers should remain aware of the complexities involved in managing such a unified system, especially as it scales. The version notes indicate that this approach is set to be showcased at KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China in September 2026, highlighting its relevance and ongoing development in the Kubernetes ecosystem.