Secure Multi-Tenant GPU Metrics in Kubernetes: A Deep Dive
In a multi-tenant Kubernetes setup, the challenge of securely managing GPU metrics is significant. Teams need to access their metrics without compromising the data of others. This is where a robust solution comes into play, allowing teams to declare their desired metrics while maintaining strict isolation and security.
The mechanism relies on three key steps: Identify, Isolate, and Deliver. First, you authenticate the caller to establish which tenant they belong to. Next, you enforce query-time isolation by restricting every query to the tenant’s namespace, ensuring that no data can be accessed outside of it. Finally, you can optionally copy a curated slice of each tenant’s metrics into their own Prometheus instance, allowing them to run dashboards and alerts against a store they fully control. A critical configuration parameter here is metricIsolation, which determines whether to collect only the series from the specified namespace.
In production, ensure that the tenant’s Prometheus is set up to accept remote writes by starting it with the –web.enable-remote-write-receiver flag. This setup is essential for the metrics to flow correctly into the tenant's Prometheus instance. Be aware that while this solution provides strong isolation, it requires careful configuration to avoid misconfigurations that could expose sensitive data.
Key takeaways
- →Implement MetricAccess to declare specific metrics for each team.
- →Use kube-rbac-proxy for authentication and authorization in a Kubernetes-native way.
- →Set `metricIsolation` to true to ensure only the namespace's series are collected.
- →Utilize prom-label-proxy to inject namespace matchers for query-time isolation.
- →Ensure the tenant's Prometheus is configured for remote-write to accept metrics.
Why it matters
In production, secure and isolated access to GPU metrics can prevent data leaks between teams, enhancing security and compliance. This setup allows teams to operate independently while maintaining control over their metrics.
Code examples
1apiVersion: observability.ethos.io/v1alpha1
2kind: MetricAccess
3metadata:
4 name: gpu-team-metrics
5 namespace: gpu-team
6spec:
7 source: gpu-team
8 metricIsolation: true # only collect this namespace's series
9 metrics:
10 - "DCGM_FI_DEV_GPU_UTIL" # exact match
11 - "container_(cpu|memory)_.*" # regex
12 - '{__name__=~"nginx_ingress_controller_.*"}' # PromQL selector
13 remoteWrite:
14 enabled: true
15 interval: "30s"
16 target:
17 type: "prometheus"
18 prometheus:
19 serviceName: "prometheus-operated"
20 servicePort: 9090
21 replicas: 2 # write to both HA replicas
22 statefulSetName: "prometheus-gpu-team"
23 extraLabels:
24 tenant: "gpu-team"
25 managed_by: "multi-tenant-proxy"curl -H "X-Tenant-Namespace: gpu-team" \
"http://prometheus-multi-tenant-proxy:8080/api/v1/query?query=DCGM_FI_DEV_GPU_UTIL"When NOT to use this
The official docs don't call out specific anti-patterns here. Use your judgment based on your scale and requirements.
Want the complete reference?
Read official docsIndustry-standard certifications built by the people behind Linux and Kubernetes. Earn the CKA — the gold standard Kubernetes administrator cert. OpsCanary readers get 30% off year-round with code OPSCANARY3.
Get CKA certified →Transforming Kubernetes: From Cloud Native to AI Native
As AI continues to evolve, so must our infrastructure. Discover how Kubernetes can adapt to support AI-native applications while avoiding common pitfalls. Learn why relying solely on simplified tools can lead to serious security issues.
Scaling AI with Kubernetes: The Role of CNCF Silver Members
As enterprises scale AI from training to inference, operational efficiency becomes critical. Kubernetes plays a vital role in managing these workloads, and the support from CNCF Silver Members enhances this infrastructure.
Unifying AI Training and Inference on Kubernetes: Lessons from China Merchants Bank
China Merchants Bank has cracked the code on unifying AI training and inference using Kubernetes. By leveraging tools like Kueue and Fluid, they efficiently manage nearly 10,000 heterogeneous accelerator cards. Dive into how this unified control plane can revolutionize your AI workflows.
Get the daily digest
One email. 5 articles. Every morning.
No spam. Unsubscribe anytime.