Building a Unified NOC Dashboard for Amazon EKS with CloudWatch
In today's cloud-native world, having a centralized view of your Kubernetes cluster's health and performance is crucial. A single-pane NOC dashboard allows you to monitor metrics in real-time, enhancing your ability to respond to incidents swiftly. By integrating Amazon CloudWatch with your Amazon EKS cluster, you can visualize critical metrics and application signals, ensuring you stay ahead of potential issues.
The process begins with the Amazon CloudWatch Observability EKS add-on, which sends Kubernetes metrics to CloudWatch using the OpenTelemetry Protocol (OTLP). You need to ensure your cluster runs Kubernetes version 1.28 or later and that you have the CloudWatch Observability EKS add-on version 6.2.0 or later installed. Once set up, you can instrument your application services for Application Signals and use PromQL to query the metrics. The builder composes PromQL chart widgets and CloudWatch metric widgets into your NOC dashboard, allowing you to validate your query manifests with a probe before executing the PutDashboard command.
In production, remember that treating FAILED metrics as blocking defects is essential. An EMPTY status should be viewed as a workload condition that requires investigation, not as a success. The queries discussed here were validated with add-on version 6.4.0-eksbuild.1, so ensure you are on a compatible version to avoid discrepancies in your dashboard metrics.
Key takeaways
- →Use the OpenTelemetry Protocol (OTLP) to send Kubernetes metrics to CloudWatch.
- →Install the CloudWatch Observability EKS add-on version 6.2.0 or later.
- →Query metrics using PromQL for effective monitoring.
- →Treat FAILED metrics as blocking defects and investigate EMPTY statuses.
- →Validate query manifests with a probe before running PutDashboard.
Why it matters
In production, having a unified view of your EKS cluster's health can significantly reduce response times to incidents, minimizing downtime and improving service reliability.
Code examples
1== retail-store-noc
2PROMQL TOTAL=19 OK=19 EMPTY=0 FAILED=0
3CLASSIC UNIQUE=28 OK=28 EMPTY=0 FAILED=0
4LOGS GROUPS=0 OK=0 FAILED=0
5BODY_SHA256=97bb1de9bb151bb47306bff70736fd73151e4d5263902408df28bf76bb796f87
6ADDON_VERSION_TESTED=v6.4.0-eksbuild.11== retail-store-application-health
2PROMQL TOTAL=0 OK=0 EMPTY=0 FAILED=0
3CLASSIC UNIQUE=29 OK=26 EMPTY=3 FAILED=0
4LOGS GROUPS=0 OK=0 FAILED=0
5EMPTY ApplicationSignals/Latency {'RemoteService': 'AWS::DynamoDB'}
6EMPTY AWS/DynamoDB/ReadThrottleEvents {'TableName': 'retail-store-carts'}
7EMPTY AWS/DynamoDB/WriteThrottleEvents {'TableName': 'retail-store-carts'}
8BODY_SHA256=af255d7f3e343bbc6cd8626416f91ab3e0afda9e07b01d6d2c84285ff6bb859bWhen NOT to use this
The official docs don't call out specific anti-patterns here. Use your judgment based on your scale and requirements.
Want the complete reference?
Read official docsIndustry-standard certifications built by the people behind Linux and Kubernetes. Earn the CKA — the gold standard Kubernetes administrator cert. OpsCanary readers get 30% off year-round with code OPSCANARY3.
Get CKA certified →Bridging the Operations Gap: Running Applications on Kubernetes
Deploying an application is just the beginning; keeping it running smoothly is where the real challenge lies. With Elastic Beanstalk's managed deployments and automatic rollback, you can ensure your application remains healthy and compliant. Dive in to learn how to leverage these features effectively.
OpenTelemetry Migration: Scaling Your Metrics Platform in Kubernetes
Migrating to OpenTelemetry can transform your metrics platform, but it’s not without its challenges. By replacing the gostatsd sidecar with an OTel Collector, you can drastically reduce data points while maintaining application compatibility. Dive in to learn how to make this transition smoothly.
Kubernetes v1.37: Embrace Native Histograms for Better Metrics
Kubernetes v1.37 brings native histograms to beta, transforming how you handle metrics. Say goodbye to static user-defined buckets and hello to dynamic, exponential buckets that offer more flexibility and precision.
Get the daily digest
One email. 5 articles. Every morning.
No spam. Unsubscribe anytime.