An incident is the worst moment to discover that you cannot trust the Network Operations Center (NOC) dashboard for your Amazon Elastic Kubernetes Service (Amazon EKS) cluster. A blank widget can mean either that the workload is healthy or that telemetry has stopped. An availability number can hide a failing service behind a quiet one. A wall display can answer none of the questions an operator is asking. Each costs response time. This post shows you what a trustworthy NOC for Amazon EKS on Amazon CloudWatch looks like and why each design choice matters.
It also shows how to have one running in your account in about 15 minutes. The challenge is structural. On an Amazon EKS cluster, the Amazon CloudWatch Observability EKS add-on sends Kubernetes metrics to CloudWatch through the OpenTelemetry Protocol (OTLP), and you query them with PromQL . AWS calls this OpenTelemetry (OTel) collection path OTel Container Insights . The application services publish golden signals through Amazon CloudWatch Application Signals .
Elastic Load Balancing , Amazon DynamoDB , Amazon Relational Database Service (Amazon RDS) , Amazon ElastiCache , and Amazon MQ publish to CloudWatch metric namespaces. No single widget type reaches all of these signals, so the NOC combines PromQL chart widgets with CloudWatch metric widgets. We validated this sample on Amazon EKS Auto Mode, but the dashboard pattern also works on standard Amazon EKS clusters with supported Linux worker nodes. For a standard cluster, use Kubernetes 1. 28 or later and install version 6. 0 or later of the Amazon CloudWatch Observability EKS add-on.
Then enable OTel Container Insights, configure EKS Pod Identity or AWS Identity and Access Management (IAM) roles for service accounts (IRSA), attach CloudWatchAgentServerPolicy to the node IAM role, and allow outbound access to CloudWatch endpoints. We validated the queries in this post with add-on v6. Rerun the probe if you use another version. Only the scale-in discussion later in the post applies specifically to Auto Mode. The complete solution lives in the sample repository on GitHub. It contains a builder that generates a 35-widget NOC and three drilldowns for the AWS retail store sample application .