Kubernetes v1.37: Embrace Native Histograms for Better Metrics
Kubernetes v1.37 introduces native histograms, a significant upgrade that addresses the limitations of classic histograms. Classic histograms require you to define static cumulative bucket boundaries, which can be cumbersome and limit your observability capabilities. Native histograms replace this with dynamic, exponential buckets, allowing for a more nuanced view of your metrics without the need for constant reconfiguration.
The implementation of native histograms is integrated directly into the shared metrics subsystem. When you enable the NativeHistograms feature gate, Kubernetes components will emit both classic and native spans, ensuring zero disruption for existing observability stacks. Key configuration parameters include BucketFactor, which controls the width of the exponential buckets (default is 1.1), and MaxBucketNumber, which caps the maximum number of buckets at 160. You can also enable scraping of native histograms and ensure classic histograms are always scraped to maintain compatibility with existing dashboards.
In production, it's crucial to remember that while transitioning to native histograms, you must set always_scrape_classic_histograms: true to avoid breaking your existing alerts and dashboards. This feature was previously in alpha in Kubernetes v1.36 and is now ready for broader use. Pay attention to the caution regarding migrating dashboards and alerts, as this can save you from headaches down the line.
Key takeaways
- →Enable native histograms by setting the NativeHistograms feature gate.
- →Configure BucketFactor to adjust the width of your histogram buckets.
- →Always set always_scrape_classic_histograms: true during your transition.
- →Utilize native histograms for more accurate metrics without static boundaries.
- →Understand the dual exposition of classic and native spans for seamless integration.
Why it matters
This enhancement allows for more precise monitoring and observability, which is crucial for maintaining performance and reliability in production environments. The ability to dynamically adjust buckets leads to better insights and faster troubleshooting.
Code examples
scrape_configs:
- job_name: 'kubernetes-apiservers'
scrape_native_histograms: true
always_scrape_classic_histograms: truecurl --insecure \
-H "Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited" \
--header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
https://localhost:6443/metrics# 1. Calculating P99 latency for a single target:
# Classic histogram (requires _bucket suffix):
histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m]))
# Native histogram (operates directly on the metric name):
histogram_quantile(0.99, rate(apiserver_request_duration_seconds[5m]))When NOT to use this
The official docs don't call out specific anti-patterns here. Use your judgment based on your scale and requirements.
Want the complete reference?
Read official docsIndustry-standard certifications built by the people behind Linux and Kubernetes. Earn the CKA — the gold standard Kubernetes administrator cert. OpsCanary readers get 30% off year-round with code OPSCANARY3.
Get CKA certified →Seamless Distributed Tracing for CI Pipelines in Kubernetes
Unlock the power of distributed tracing in your CI pipelines without modifying a single workflow file. By leveraging an OpenTelemetry Collector with GitHub webhooks, you can transform workflow events into actionable OTLP spans effortlessly.
OpenTelemetry Graduation: What Comes Next for Kubernetes Monitoring
OpenTelemetry's graduation marks a pivotal moment in observability, merging tracing, metrics, and logs into a unified framework. With standardized APIs and a robust Collector, it simplifies monitoring in Kubernetes environments. This article dives into what this means for your production systems.
Unlocking Observability in Kubernetes: From Metrics to Meaning
Observability is crucial for managing complex Kubernetes environments. By leveraging metrics, logs, and traces, you can transform raw data into actionable insights. Learn how to implement these signals effectively in your production systems.
Get the daily digest
One email. 5 articles. Every morning.
No spam. Unsubscribe anytime.