kubernetesai workloadsPractitioner

Mastering GPU Auto-Repair with Amazon ECS Managed Instances

5 min read AWS Containers BlogNov 12, 2026Reviewed for accuracy

Practitioner — Hands-on experience recommended

In production environments, GPU failures can lead to significant disruptions. Amazon ECS Managed Instances address this issue with built-in GPU health monitoring and automatic repair powered by NVIDIA Data Center GPU Manager (DCGM). This capability allows your applications to maintain high availability and performance, even in the face of hardware issues.

When a critical GPU failure occurs, DCGM reports an XID error code, prompting ECS to take action. The instance is marked as IMPAIRED and set to DRAINING status, preventing new tasks from being scheduled on it. ECS then provisions a healthy replacement instance, gracefully stopping existing tasks on the impaired instance while respecting their configured stopTimeout. Once all tasks are stopped, ECS terminates the impaired instance, seamlessly transitioning your workloads to the new instance.

In practice, you need to ensure that GPU auto-repair is enabled through the autoRepairConfiguration.actionsStatus parameter, which defaults to ENABLED. Be cautious when testing this feature; the XID error is synthetic but triggers the real drain-and-replacement workflow. Always run such tests in a dedicated test cluster or during maintenance windows to avoid impacting production workloads.

Key takeaways

→Enable GPU auto-repair by configuring `autoRepairConfiguration.actionsStatus` to ENABLED.
→Understand that DCGM monitors GPU health and reports critical failures using XID error codes.
→Recognize that ECS uses a start-before-stop pattern to replace impaired instances without disrupting tasks.
→Test the auto-repair feature in a controlled environment to prevent unintended production issues.

Why it matters

Automatically recovering from GPU failures minimizes downtime and keeps your applications running smoothly, which is crucial for maintaining service reliability and performance in production environments.

Code examples

JSON

1{
2  "version": "0",
3  "id": "<event-id>",
4  "detail-type": "ECS Container Instance Health Change",
5  "source": "aws.ecs",
6  "account": "<account-id>",
7  "time": "<event-time>",
8  "region": "<region>",
9  "resources": [
10    "arn:aws:ecs:<region>:<account-id>:container-instance/<cluster-name>/<container-instance-id>"
11  ],
12  "detail": {
13    "clusterArn": "arn:aws:ecs:<region>:<account-id>:cluster/<cluster-name>",
14    "containerInstanceArn": "arn:aws:ecs:<region>:<account-id>:container-instance/<cluster-name>/<container-instance-id>",
15    "capacityProviderName": "<capacity-provider-name>",
16    "ec2InstanceId": "<instance-id>",
17    "overallStatus": "IMPAIRED",
18    "healthChecks": [
19      {
20        "type": "ACCELERATED_COMPUTE",
21        "status": "IMPAIRED",
22        "lastStatusChange": "<last-status-change-time>",
23        "lastUpdated": "<last-updated-time>",
24        "statusReason": "XID_79"
25      }
26    ]
27  }
28}

When NOT to use this

The official docs don't call out specific anti-patterns here. Use your judgment based on your scale and requirements.

Want the complete reference?

Read official docs

Test what you just learned

Quiz questions written from this article

Take the quiz →

Linux FoundationSponsor

Industry-standard certifications built by the people behind Linux and Kubernetes. Earn the CKA — the gold standard Kubernetes administrator cert. OpsCanary readers get 30% off year-round with code OPSCANARY3.

Get CKA certified →

Mastering GPU Auto-Repair with Amazon ECS Managed Instances

Key takeaways

Why it matters

Code examples

When NOT to use this

More on this topic

Mastering Gang Scheduling and Topology-Aware Placement in Kubernetes for AI Training

Meta's GitOps Revolution: From Hours to Minutes in AI Infrastructure Deployment

Vultr Joins CNCF as Platinum Member: What This Means for AI and Kubernetes