Building a Cluster-Aware AI Agent with Kubernetes and GitOps

In today's fast-paced tech landscape, deploying AI agents that can operate within a Kubernetes cluster is essential. These agents can enhance application capabilities by leveraging Large Language Models (LLMs) locally, reducing latency and dependency on external services. By integrating GitOps practices with tools like Argo CD, you can ensure that your AI deployments are not only efficient but also maintainable and scalable.

The architecture consists of a CI/CD chain and a Kubernetes runtime. On the runtime side, an Ollama pod serves a local Mistral 7B model, exposing a REST API on port 11434. A FastAPI pod provides the agent's HTTP API and chat UI on port 8000, while a PersistentVolumeClaim holds the model weights. A dedicated ServiceAccount in the FastAPI pod has a ClusterRole that allows only read operations. When you push changes to the application source in Git, GitHub Actions builds a multi-architecture image tagged with the 7-character commit SHA. Argo CD Image Updater checks Docker Hub every two minutes for new tags, commits the new tag back into the repository's kustomization.yaml, and Argo CD reconciles the cluster to deploy the updates.

In production, be mindful of the configuration parameters, such as allowTags, which uses a regex for allowed tags, and updateStrategy, which defaults to newest-build. These settings help control how your images are updated and ensure that only valid tags are deployed. However, be cautious about giving your AI agent write access to the cluster. An agent that can delete pods based on its own reasoning is a production incident waiting to happen, as hallucinations combined with write access can lead to disastrous outcomes.