When you run GPU inference at scale, hardware failures are a matter of when, not if. Double-bit ECC memory errors can crash running applications, NVLink failures can break multi-GPU communication, and a GPU falling off the PCIe bus makes it completely unreachable. An unhealthy GPU can stall workloads and leave expensive accelerator capacity idle until someone detects the failure and replaces the host. As GPU fleets grow, detecting these failures and recovering workloads quickly becomes an important part of operating a reliable inference platform.
On self-managed GPU fleets, teams typically build monitoring and remediation workflows around NVIDIA GPU health signals. This can involve custom health-check scripts that poll NVIDIA SMI, Amazon CloudWatch alarms , an AWS Lambda function that deregisters the instance from the load balancer, and a manual or scripted instance refresh. Each step adds operational complexity and latency, and introduces new failure modes. Amazon Elastic Container Service (Amazon ECS) Managed Instances provides built-in GPU health monitoring and automatic repair powered by NVIDIA Data Center GPU Manager (DCGM).
When Amazon ECS detects a critical GPU hardware failure, it marks the instance impaired, sets it to draining so no new tasks land on it, provisions a healthy replacement, and terminates the impaired instance after all tasks have stopped. You don’t need to change application code or take any operator action. GPU auto-repair is enabled by default for Amazon ECS Managed Instances. In this post, we use an asynchronous GPU inference pipeline as a running example to show how GPU auto-repair works in practice.
We cover which GPU failures trigger instance replacement, how Amazon ECS performs the repair, how to observe health changes through Amazon EventBridge and Amazon CloudWatch, and how to safely validate the complete workflow using DCGM fault injection. Operating GPU inference involves more than running containers. On self-managed fleets, teams are responsible for provisioning instances, maintaining AMIs and GPU drivers, applying security updates, monitoring GPU health, and building remediation workflows when hardware fails. ECS Managed Instances handles this infrastructure lifecycle for you.
