Building a Unified NOC Dashboard for Amazon EKS with CloudWatch
In today's cloud-native world, having a centralized view of your Kubernetes cluster's health and performance is crucial. A single-pane NOC dashboard allows you to monitor metrics in real-time, enhancing your ability to respond to incidents swiftly. By integrating Amazon CloudWatch with your Amazon EKS cluster, you can visualize critical metrics and application signals, ensuring you stay ahead of potential issues.
The process begins with the Amazon CloudWatch Observability EKS add-on, which sends Kubernetes metrics to CloudWatch using the OpenTelemetry Protocol (OTLP). You need to ensure your cluster runs Kubernetes version 1.28 or later and that you have the CloudWatch Observability EKS add-on version 6.2.0 or later installed. Once set up, you can instrument your application services for Application Signals and use PromQL to query the metrics. The builder composes PromQL chart widgets and CloudWatch metric widgets into your NOC dashboard, allowing you to validate your query manifests with a probe before executing the PutDashboard command.
In production, remember that treating FAILED metrics as blocking defects is essential. An EMPTY status should be viewed as a workload condition that requires investigation, not as a success. The queries discussed here were validated with add-on version 6.4.0-eksbuild.1, so ensure you are on a compatible version to avoid discrepancies in your dashboard metrics.
Key takeaways
- →Use the OpenTelemetry Protocol (OTLP) to send Kubernetes metrics to CloudWatch.
- →Install the CloudWatch Observability EKS add-on version 6.2.0 or later.
- →Query metrics using PromQL for effective monitoring.
- →Treat FAILED metrics as blocking defects and investigate EMPTY statuses.
- →Validate query manifests with a probe before running PutDashboard.
Why it matters
In production, having a unified view of your EKS cluster's health can significantly reduce response times to incidents, minimizing downtime and improving service reliability.
Code examples
1== retail-store-noc
2PROMQL TOTAL=19 OK=19 EMPTY=0 FAILED=0
3CLASSIC UNIQUE=28 OK=28 EMPTY=0 FAILED=0
4LOGS GROUPS=0 OK=0 FAILED=0
5BODY_SHA256=97bb1de9bb151bb47306bff70736fd73151e4d5263902408df28bf76bb796f87
6ADDON_VERSION_TESTED=v6.4.0-eksbuild.11== retail-store-application-health
2PROMQL TOTAL=0 OK=0 EMPTY=0 FAILED=0
3CLASSIC UNIQUE=29 OK=26 EMPTY=3 FAILED=0
4LOGS GROUPS=0 OK=0 FAILED=0
5EMPTY ApplicationSignals/Latency {'RemoteService': 'AWS::DynamoDB'}
6EMPTY AWS/DynamoDB/ReadThrottleEvents {'TableName': 'retail-store-carts'}
7EMPTY AWS/DynamoDB/WriteThrottleEvents {'TableName': 'retail-store-carts'}
8BODY_SHA256=af255d7f3e343bbc6cd8626416f91ab3e0afda9e07b01d6d2c84285ff6bb859bWhen NOT to use this
The official docs don't call out specific anti-patterns here. Use your judgment based on your scale and requirements.
Want the complete reference?
Read official docsIndustry-standard certifications built by the people behind Linux and Kubernetes. Earn the CKA — the gold standard Kubernetes administrator cert. OpsCanary readers get 30% off year-round with code OPSCANARY3.
Get CKA certified →Unlocking Kubernetes: The Workload Conformance Working Group Explained
The Workload Conformance Working Group is setting a new standard for Kubernetes workloads, addressing a critical gap in runtime behavior. This initiative will deliver objective tests for workload behavior, ensuring consistency across Kubernetes versions.
Measuring Developer Experience on EKS: Unlocking DORA Metrics
Want to elevate your developer experience on Amazon EKS? Focus on DORA metrics like Deployment Frequency and Change Failure Rate to gain actionable insights. This article dives into how to connect your CI/CD events for effective measurement.
Mastering Observability for AI Agents on Amazon EKS
Observability is crucial for understanding AI agent performance and costs in production. By leveraging LLM call traces and tool call spans, you can gain insights into your agents' behavior. This article dives into practical patterns for implementing observability on Amazon EKS.
Get the daily digest
One email. 5 articles. Every morning.
No spam. Unsubscribe anytime.