AI & GPU Workloads
20 articles from official documentation
Efficient GPU Batch Inference on ECS: Scale to Zero
Unlock the power of GPU batch inference on Amazon ECS with zero idle costs. Learn how to leverage Amazon SQS for job buffering and AWS Application Auto Scaling to optimize your resource usage.
- →Utilize Amazon SQS to buffer inference requests, ensuring durability across instance lifecycle events.
- →Set TRANSFORMERS_OFFLINE to 1 for fully air-gapped container operation after image pull.
Subaru's Kubernetes Revolution: 60x Faster AI Development
Subaru has transformed its AI development process using cloud-native infrastructure on Kubernetes, slashing container image pull times from three hours to just three minutes. Discover how they leveraged Envoy Gateway and GitOps practices to achieve this remarkable efficiency.
- →Leverage Envoy Gateway and MetalLB to optimize Kubernetes for AI workloads.
- →Adopt GitOps with Argo CD and Helmfile to standardize application delivery.
How Japan's 250% Training Surge is Shaping Global AI Talent
Japan's explosive growth in AI training is not just a regional phenomenon—it's influencing global talent dynamics. The Kubestronaut program is at the forefront, certifying essential skills for managing AI infrastructure and workloads.
- →Understand the Kubestronaut program's role in certifying AI infrastructure skills.
- →Aim for Golden Kubestronaut status by completing all CNCF certifications and obtaining LFCS.
Supercharge Your AI Workflows with Kubeflow's Latest Innovations
Kubeflow is evolving, and its latest features are game-changers for AI development. With Kale, you can turn Jupyter notebooks into production-ready pipelines without writing any KFP SDK code. This article dives into how these innovations can streamline your AI workflows.
- →Utilize Kale to convert Jupyter notebooks into production-ready pipelines effortlessly.
- →Leverage the unified Kubeflow SDK for managing end-to-end AI workloads.
AI Infra SIG: Elevating Kubernetes for AI Workloads
The launch of the AI Infra SIG under the CNCF Japan chapter is a game changer for optimizing AI workloads on Kubernetes. This initiative focuses on best practices and introduces new capabilities for AI-native infrastructure. Don't miss the call for speakers to shape the future of AI in cloud-native environments.
- →Engage with the AI Infra SIG to learn best practices for AI workloads.
- →Explore AI readiness initiatives to enhance your Kubernetes infrastructure.
Harnessing Community-Driven AI with Kubernetes: The Future is Open
The future of AI is being shaped by community-driven, open-source solutions. With tools like the NVIDIA GPU Dynamic Resource Allocation Driver, Kubernetes is evolving to meet the demands of large AI clusters.
- →Utilize the NVIDIA GPU Dynamic Resource Allocation Driver for real-time GPU allocation.
- →Implement the KAI Scheduler to manage complex scheduling demands in AI workloads.
Harnessing Velero for AI-Ready Kubernetes Infrastructure
As cloud-native technologies evolve, the need for robust backup solutions becomes critical. Velero offers a Kubernetes-native platform that safeguards AI workflows and cluster states, ensuring disaster recovery and seamless migrations.
- →Utilize Velero to protect cluster state and persistent data in Kubernetes.
- →Safeguard AI workflows to ensure disaster recovery capabilities.
Streamlining AI/ML Workloads: Headlamp Plugin for Kubeflow on Kubernetes
Managing AI and ML workloads can be complex, but the Headlamp plugin for Kubeflow simplifies this process. It interfaces directly with the Kubernetes API server to provide real-time insights into Pod conditions and failure reasons across namespaces.
- →Utilize the Headlamp plugin to access real-time insights from the Kubernetes API server.
- →Monitor Pod conditions and Kubernetes failure reasons across namespaces effortlessly.
Navigating Kubernetes Open Source Maintainership in the Age of AI
AI is reshaping how we contribute to open source projects, but it comes with its own set of challenges. Kubernetes has established a clear AI policy that mandates transparency and human accountability in contributions. Understanding these guidelines is crucial for maintainers and contributors alike.
- →Disclose AI tool usage in pull requests to maintain transparency.
- →Verify AI-generated changes through thorough code review and testing.
Building a Cluster-Aware AI Agent with Kubernetes and GitOps
Unlock the potential of AI in your Kubernetes cluster with a robust GitOps workflow. This article dives into using Ollama to serve local LLMs and Argo CD to automate deployments, ensuring your AI agent is always up-to-date.
- →Utilize Ollama to serve local LLMs and reduce latency.
- →Implement Argo CD Image Updater for automated deployment of new model versions.
Unifying AI Workloads: KubeCon, OpenInfra, and PyTorch Conference in China
Discover how the convergence of KubeCon, OpenInfra Summit, and PyTorch Conference in China is set to revolutionize AI workloads. By integrating Kubernetes orchestration with OpenInfra's infrastructure and PyTorch's AI frameworks, organizations can achieve scalable and reliable AI solutions.
- →Leverage the integration of OpenInfra for optimized infrastructure.
- →Utilize Kubernetes for effective orchestration of AI workloads.
Mastering Geo-Distributed AI Operations with k0smos
Unlock the potential of geo-distributed AI infrastructure with the k0smos stack. This powerful setup leverages k0s and k0smotron to deploy isolated control planes, streamlining operations across multiple clusters.
- →Leverage k0s for a lightweight, zero-dependency Kubernetes distribution.
- →Utilize k0smotron to deploy isolated, versioned control planes efficiently.
Engineering AI at Scale: Kubernetes for the Next Generation
AI workloads are fundamentally different from traditional microservices, and Kubernetes is evolving to meet these challenges. Discover how the Kubernetes AI Conformance program and Dynamic Resource Allocation can help you scale AI applications effectively.
- →Utilize the Kubernetes AI Conformance program to ensure interoperability across environments.
- →Implement Dynamic Resource Allocation to efficiently manage specialized hardware for AI workloads.
Achieving 30-Second LLM Cold Starts on Kubernetes with Fluid
Cold starts can cripple application performance, especially for large language models (LLMs). Discover how NetEase Games leveraged Fluid to automate runtime deployment and optimize cache management, achieving impressive 30-second cold starts on Kubernetes.
- →Leverage Fluid for automated runtime deployment and lifecycle management.
- →Utilize HPA and KEDA for cache elasticity to optimize resource scaling.
Streamline AI Workloads with Kubernetes Dynamic Resource Allocation on AWS
Simplifying AI infrastructure is crucial for efficiency and performance. With Kubernetes Dynamic Resource Allocation (DRA), you can manage AWS Trainium and Elastic Fabric Adapter devices seamlessly. This article dives into how DRA transforms resource management in Kubernetes.
- →Utilize ResourceClaimTemplates to define policies for workload patterns.
- →Leverage ResourceSlices to advertise available EFA and Neuron devices to the scheduler.
How KubeStellar Achieved 81% PR Acceptance with AI Agents
KubeStellar is revolutionizing how we approach pull requests by integrating AI coding agents into the workflow. By externalizing preferences in CLAUDE.md and measuring acceptance rates with auto-qa-tuning.json, they’ve reached an impressive 81% PR acceptance rate. Dive in to discover how this model can transform your Kubernetes projects.
- →Utilize CLAUDE.md to externalize pull request conventions.
- →Log PR acceptance rates with auto-qa-tuning.json to measure performance.
Cloud Custodian: Governance for the AI Era
As AI agents increasingly manage cloud infrastructure, effective governance becomes critical. Cloud Custodian offers automated guardrails that enforce best practices in real-time, ensuring your resources remain efficient and secure.
- →Implement automated guardrails to manage AI-generated resources effectively.
- →Utilize declarative policies to describe and enforce desired states of cloud resources.
Benchmarking AI Retrieval Strategies for Kubernetes Bug Fixes
In the vast landscape of Kubernetes, fixing bugs can be a daunting task. This article explores how different AI agent retrieval strategies—RAG, Hybrid, and Local Only—impact the effectiveness of bug fixes in a multi-million-line codebase.
- →Understand the differences between RAG, Hybrid, and Local Only strategies for bug fixes.
- →Leverage RAG's hybrid retrieval for keyword matching and semantic search to enhance fix accuracy.
Accelerate AI Model Distribution with Dragonfly's P2P Magic
Tired of slow model downloads? Dragonfly’s peer-to-peer acceleration can reduce your origin traffic by 99.5%. Discover how it splits files and shares them across nodes for lightning-fast distribution.
- →Leverage P2P to reduce model download times dramatically.
- →Configure repository types to optimize your downloads.
Deploying Generative AI at the Edge with EKS Hybrid Nodes and NVIDIA DGX
Unlock the power of generative AI at the edge with Amazon EKS Hybrid Nodes and NVIDIA DGX. This setup allows you to connect on-premises infrastructure directly to the EKS control plane, ensuring low-latency AI services. Learn how to configure your environment for optimal performance.
- →Connect on-premises infrastructure to Amazon EKS using EKS Hybrid Nodes for low-latency AI services.
- →Deploy NVIDIA DGX Spark as a hybrid node to optimize edge AI deployment.
Unified observability — logs, uptime monitoring, and on-call in one place. Used by 50,000+ engineering teams to ship faster and sleep better.
Try Better Stack free →Get the daily digest
One email. 5 articles. Every morning.
No spam. Unsubscribe anytime.