Secure Multi-Tenant GPU Metrics in Kubernetes: A Deep Dive
In a multi-tenant Kubernetes setup, the challenge of securely managing GPU metrics is significant. Teams need to access their metrics without compromising the data of others. This is where a robust solution comes into play, allowing teams to declare their desired metrics while maintaining strict isolation and security.
The mechanism relies on three key steps: Identify, Isolate, and Deliver. First, you authenticate the caller to establish which tenant they belong to. Next, you enforce query-time isolation by restricting every query to the tenant’s namespace, ensuring that no data can be accessed outside of it. Finally, you can optionally copy a curated slice of each tenant’s metrics into their own Prometheus instance, allowing them to run dashboards and alerts against a store they fully control. A critical configuration parameter here is metricIsolation, which determines whether to collect only the series from the specified namespace.
In production, ensure that the tenant’s Prometheus is set up to accept remote writes by starting it with the –web.enable-remote-write-receiver flag. This setup is essential for the metrics to flow correctly into the tenant's Prometheus instance. Be aware that while this solution provides strong isolation, it requires careful configuration to avoid misconfigurations that could expose sensitive data.
Key takeaways
- →Implement MetricAccess to declare specific metrics for each team.
- →Use kube-rbac-proxy for authentication and authorization in a Kubernetes-native way.
- →Set `metricIsolation` to true to ensure only the namespace's series are collected.
- →Utilize prom-label-proxy to inject namespace matchers for query-time isolation.
- →Ensure the tenant's Prometheus is configured for remote-write to accept metrics.
Why it matters
In production, secure and isolated access to GPU metrics can prevent data leaks between teams, enhancing security and compliance. This setup allows teams to operate independently while maintaining control over their metrics.
Code examples
1apiVersion: observability.ethos.io/v1alpha1
2kind: MetricAccess
3metadata:
4 name: gpu-team-metrics
5 namespace: gpu-team
6spec:
7 source: gpu-team
8 metricIsolation: true # only collect this namespace's series
9 metrics:
10 - "DCGM_FI_DEV_GPU_UTIL" # exact match
11 - "container_(cpu|memory)_.*" # regex
12 - '{__name__=~"nginx_ingress_controller_.*"}' # PromQL selector
13 remoteWrite:
14 enabled: true
15 interval: "30s"
16 target:
17 type: "prometheus"
18 prometheus:
19 serviceName: "prometheus-operated"
20 servicePort: 9090
21 replicas: 2 # write to both HA replicas
22 statefulSetName: "prometheus-gpu-team"
23 extraLabels:
24 tenant: "gpu-team"
25 managed_by: "multi-tenant-proxy"curl -H "X-Tenant-Namespace: gpu-team" \
"http://prometheus-multi-tenant-proxy:8080/api/v1/query?query=DCGM_FI_DEV_GPU_UTIL"When NOT to use this
The official docs don't call out specific anti-patterns here. Use your judgment based on your scale and requirements.
Want the complete reference?
Read official docsIndustry-standard certifications built by the people behind Linux and Kubernetes. Earn the CKA — the gold standard Kubernetes administrator cert. OpsCanary readers get 30% off year-round with code OPSCANARY3.
Get CKA certified →Mastering Multi-Tenant GPU Kubernetes with Kamaji and HAMi
Unlock the power of multi-tenant GPU workloads in Kubernetes using Kamaji and HAMi. This setup allows you to efficiently allocate GPU resources across multiple tenants, leveraging simulated hardware for testing and development. Dive into the specifics of configuration and deployment to streamline your GPU management.
Maximize Your AI SRE Evaluation with Contextual Awareness
Before diving into AI SRE evaluations, ensure your existing agent has the right context. By leveraging Radar's read-only MCP tools, you can access critical insights like current failures and change timelines that raw kubectl can't provide. This context is essential for effective decision-making.
Unlocking BackstageCon: Insights for KubeCon + CloudNativeCon 2026
BackstageCon is set to be a pivotal gathering for Backstage enthusiasts at KubeCon + CloudNativeCon 2026. Dive into AI features and the Model Context Protocol that are reshaping software development workflows. This is your chance to connect with the community and deepen your understanding of Backstage.
Get the daily digest
One email. 5 articles. Every morning.
No spam. Unsubscribe anytime.