Running a 35B-class MoE on consumer hardware is bounded not by compute but by weight memory: at 4-bit the weights still take 19.5GB, and sparsity shrinks the compute per token, not the bytes that must be held. Offloading expert weights to SSD is the obvious idea, but naive offloading does not work — which experts layer N+1 needs is only known after layer N finishes, so the reads can never start early enough to hide behind compute. A paper released September 16 by the Edge0 team offers a remarkably thorough answer (arXiv:2609.18063).

Prediction IS the Route

The core component is called the prerouter: a per-layer head predicts the next layer's routing one token ahead, and the prediction is consumed as the routing itself — the staged expert set equals the routed set, nothing is dropped. There is a subtle architectural choice here: the prediction head is not a side-channel guesser; its output literally defines the route. A reader in the HF paper page comments asked the sharp question: when a prediction is wrong, do you stall the decode to fetch the right experts, or push forward and eat the quality hit? Author Yu Lin's reply was one sentence: "we use prediction result as routing, so it is 100% accuracy." That is not evasion — it is redefining the problem. Routing is no longer the thing being predicted; it is the behavior defined by the prediction. The cost is absorbed by Recover-LoRA: an unmerged LoRA trained on the student path pays back the quality lost to int4 quantization and routing replacement, while the base stays frozen and read-only — one base serves multiple adapter sets.

The Numbers

From the README benchmark tables (OpenCompass, identical settings vs the fp16 base):

  • Speed: on a single Mac mini M4 Pro 24GB, edge0-35b decodes at 14.9–17.7 tok/s inside 2.9 GiB peak active memory; edge0-8b hits 23.9–25.3 tok/s in 1.0 GiB
  • Quality: five-benchmark average 79.2 vs 83.2 for the fp16 base — officially reported as a 3.9-point average loss, 2.8 for the 8B tier; on MMLU-Pro the edge0-8b actually beats its base (70.1 vs 65.8)
  • Throughput gain: the prerouter delivers up to +59% decode throughput, and the gain grows with storage latency, model size, and routed width K
  • Footprint: 4-bit checkpoints are ~23GB (35B) and ~4.2GB (8B); expert weights are mmapped and read on demand, never preloaded into RAM

The Other Half of the Memory Wall

The "other half" in the paper's title is the point: for the past two years the industry has aimed its memory-wall anxiety at attention and KV caches, while the weight side of MoE is the harder constraint on consumer devices — the title is the manifesto. Getting 14.9–17.7 tok/s from a 35B tier on Apple Silicon turns local deployment of 30B+ models from a demo video into a daily tool. The framework, checkpoints, and adapters are all open source (Apache-2.0); the base models are Qwen3.6-35B-A3B and the Ling 3.0 bailing hybrid respectively, and edge0 serve starts an OpenAI-compatible server in one command. The current MLX backend only supports macOS on Apple Silicon (M1–M4); a CUDA backend is on the roadmap. For ordinary users the barrier is no longer whether you can afford a GPU — it is whether you can spare 23GB of SSD.

So what? When the routing prediction itself becomes the route, "how accurate is the prediction" stops being an engineering question and becomes a definitional one — whether this pattern shows up in larger MoE serving stacks is worth watching.