kubernetesstoragePractitioner

Deploying vLLM in Kubernetes: A Practical Guide

5 min read CNCF BlogJul 16, 2026Reviewed for accuracy

Practitioner — Hands-on experience recommended

In today's AI landscape, deploying large language models (LLMs) efficiently is crucial. vLLM is a high-performance, open-source inference engine designed to handle many concurrent requests in a Kubernetes cluster. This setup not only allows for scalable deployments but also ensures that your models are served with minimal latency, addressing the common pain points of managing LLMs in production.

The architecture relies on Kubernetes for orchestration and LINSTOR for persistent storage. By using the LINSTOR Container Storage Interface (CSI) driver, you can create a PersistentVolumeClaim (PVC) that provisions a thin-provisioned LVM volume with two replicas. This setup guarantees that your model weights are cached in the specified mountPath, /root/.cache/huggingface, allowing for quick access and avoiding redundant downloads. You need to set the HF_TOKEN environment variable with your Hugging Face access token and configure the gpu-memory-utilization parameter to control memory reservation effectively. A value of 0.80 is recommended to prevent startup failures due to memory overcommitment.

In production, remember to replace the placeholder token in your configurations with your actual Hugging Face access token. Be cautious with the gpu-memory-utilization flag; it defaults to 92% of available memory, which can lead to issues if not adjusted. Scaling the deployment to zero is a useful strategy to keep your PVC and cached model weights intact, allowing for seamless scaling back up without the need to re-download models. Overall, vLLM provides a robust solution for serving LLMs in Kubernetes, but attention to detail in configuration is key to a smooth deployment.

Key takeaways

→Use LINSTOR for persistent storage to avoid repeated model downloads.
→Set the HF_TOKEN environment variable with your actual Hugging Face token.
→Adjust gpu-memory-utilization to 0.80 to prevent startup failures.
→Scale the deployment to zero to maintain PVC and cached model weights.
→Ensure the StorageClass name matches your environment before applying configurations.

Why it matters

Efficiently deploying LLMs can significantly reduce latency and operational overhead, making your applications more responsive and cost-effective. This is especially important as demand for AI-driven features grows.

Code examples

YAML

1cat
2<<EOF
3|
4kubectl apply -f -
5apiVersion: v1
6kind: PersistentVolumeClaim
7metadata:
8  name: vllm-models
9spec:
10  storageClassName: linstor-csi-lvm-thin-r2
11  accessModes:
12    - ReadWriteOnce
13  volumeMode: Filesystem
14  resources:
15    requests:
16      storage: 50Gi
17---
18apiVersion: v1
19kind: Secret
20metadata:
21  name: hf-token-secret
22type: Opaque
23stringData:
24  token: "REPLACE_WITH_YOUR_TOKEN"
25EOF

YAML

1VLLM_IMAGE
2=
3public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo:latest
4cat
5<<EOF
6|
7kubectl apply -f -
8apiVersion: apps/v1
9kind: Deployment
10metadata:
11  name: vllm-server
12spec:
13  replicas: 1
14  selector:
15    matchLabels:
16      app.kubernetes.io/name: vllm
17  template:
18    metadata:
19      labels:
20        app.kubernetes.io/name: vllm
21    spec:
22      containers:
23      - name: vllm
24        image: $VLLM_IMAGE
25        command: ["/bin/sh", "-c"]
26        args: [
27          "vllm serve meta-llama/Llama-3.2-1B-Instruct --gpu-memory-utilization 0.80"
28        ]
29        env:
30        - name: HF_TOKEN
31          valueFrom:
32            secretKeyRef:
33              name: hf-token-secret
34              key: token
35        ports:
36          - containerPort: 8000
37        volumeMounts:
38          - name: llama-storage
39            mountPath: /root/.cache/huggingface
40      volumes:
41      - name: llama-storage
42        persistentVolumeClaim:
43          claimName: vllm-models
44---
45apiVersion: v1
46kind: Service
47metadata:
48  name: vllm-server
49spec:
50  selector:
51    app.kubernetes.io/name: vllm
52  ports:
53  - protocol: TCP
54    port: 8000
55    targetPort: 8000
56  type: ClusterIP
57EOF

YAML

1cat
2<<EOF
3|
4kubectl apply -f -
5apiVersion: v1
6kind: Pod
7metadata:
8  name: curl-client
9  namespace: default
10spec:
11  containers:
12  - name: curl
13    image: curlimages/curl:latest
14    command: ["sleep", "infinity"]
15  restartPolicy: Never
16EOF