Kubernetes v1.37: Embrace Native Histograms for Better Metrics
Kubernetes v1.37 introduces native histograms, a significant upgrade that addresses the limitations of classic histograms. Classic histograms require you to define static cumulative bucket boundaries, which can be cumbersome and limit your observability capabilities. Native histograms replace this with dynamic, exponential buckets, allowing for a more nuanced view of your metrics without the need for constant reconfiguration.
The implementation of native histograms is integrated directly into the shared metrics subsystem. When you enable the NativeHistograms feature gate, Kubernetes components will emit both classic and native spans, ensuring zero disruption for existing observability stacks. Key configuration parameters include BucketFactor, which controls the width of the exponential buckets (default is 1.1), and MaxBucketNumber, which caps the maximum number of buckets at 160. You can also enable scraping of native histograms and ensure classic histograms are always scraped to maintain compatibility with existing dashboards.
In production, it's crucial to remember that while transitioning to native histograms, you must set always_scrape_classic_histograms: true to avoid breaking your existing alerts and dashboards. This feature was previously in alpha in Kubernetes v1.36 and is now ready for broader use. Pay attention to the caution regarding migrating dashboards and alerts, as this can save you from headaches down the line.
Key takeaways
- →Enable native histograms by setting the NativeHistograms feature gate.
- →Configure BucketFactor to adjust the width of your histogram buckets.
- →Always set always_scrape_classic_histograms: true during your transition.
- →Utilize native histograms for more accurate metrics without static boundaries.
- →Understand the dual exposition of classic and native spans for seamless integration.
Why it matters
This enhancement allows for more precise monitoring and observability, which is crucial for maintaining performance and reliability in production environments. The ability to dynamically adjust buckets leads to better insights and faster troubleshooting.
Code examples
scrape_configs:
- job_name: 'kubernetes-apiservers'
scrape_native_histograms: true
always_scrape_classic_histograms: truecurl --insecure \
-H "Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited" \
--header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
https://localhost:6443/metrics# 1. Calculating P99 latency for a single target:
# Classic histogram (requires _bucket suffix):
histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m]))
# Native histogram (operates directly on the metric name):
histogram_quantile(0.99, rate(apiserver_request_duration_seconds[5m]))When NOT to use this
The official docs don't call out specific anti-patterns here. Use your judgment based on your scale and requirements.
Want the complete reference?
Read official docsIndustry-standard certifications built by the people behind Linux and Kubernetes. Earn the CKA — the gold standard Kubernetes administrator cert. OpsCanary readers get 30% off year-round with code OPSCANARY3.
Get CKA certified →Unlocking Kubernetes: The Workload Conformance Working Group Explained
The Workload Conformance Working Group is setting a new standard for Kubernetes workloads, addressing a critical gap in runtime behavior. This initiative will deliver objective tests for workload behavior, ensuring consistency across Kubernetes versions.
Measuring Developer Experience on EKS: Unlocking DORA Metrics
Want to elevate your developer experience on Amazon EKS? Focus on DORA metrics like Deployment Frequency and Change Failure Rate to gain actionable insights. This article dives into how to connect your CI/CD events for effective measurement.
Mastering Observability for AI Agents on Amazon EKS
Observability is crucial for understanding AI agent performance and costs in production. By leveraging LLM call traces and tool call spans, you can gain insights into your agents' behavior. This article dives into practical patterns for implementing observability on Amazon EKS.
Get the daily digest
One email. 5 articles. Every morning.
No spam. Unsubscribe anytime.