Why MoE inference has been burning FLOPs it never needed

Mixture-of-Experts has become the default way to scale LLM parameters — Qwen3, Mixtral, DeepSeek, GLM-5 all lean on it to push total parameters into the hundreds of billions while activating only a slice at any moment. But the new paper on arXiv 2609.05228, titled ACE, calls out an awkward fact: with fixed top-k routing, every token activates the same number of expert slots, regardless of whether it actually needs them. The paper dissects this into an engineering problem and proposes a training-free, calibration-free, checkpoint-preserving framework for token-adaptive expert skipping.

How ACE decides what to skip: GSP + RCR

The core idea is "two independent views agree, then skip." The first view is Global Spectral Proxy (GSP): it treats the coupled gate, up, and down projections of each expert as one block and estimates the expert's real "global transformation capacity" under RMSNorm scaling — in effect asking how far, and how stably, the expert can push the input vector. That statistic is computed offline; at inference it is just a lookup.

The second view is Router-Conditioned Refinement (RCR): it projects de-centered router weights into expert-specific direction prototypes and then evaluates the expert's response on the current token along routing-preferred directions. Only when both views flag an expert slot as low-contribution does ACE skip it — and the top-1 expert is always retained.

Three MoE LLMs, eight benchmarks — fewer FLOPs, better numbers

The paper reports experiments across three MoE-based LLMs and eight benchmarks. The headline numbers (taken directly from the arXiv abstract):

  • On a Qwen-family 35B-A3B MoE, at a 50% skipping ratio, the paper reports WikiText-2 perplexity dropping 7.96% and downstream average accuracy rising 4.15 percentage points over the strongest competing method
  • The advantage over static and dynamic baselines widens at more aggressive skipping rates
  • All expert statistics are computed offline; inference time is just table lookups and lightweight scalar operations — no extra training, no calibration data, no checkpoint modifications

In other words: it is a "runtime efficiency shell" that wraps an already-trained MoE, without retraining, fine-tuning, or touching the original weights.

Why has nobody done "skip vs. don't skip" cleanly before

Existing expert-sparsification routes all have visible pain points. Router-confidence methods get fooled by softmax bias in the gating distribution. Calibration-data methods need a representative held-out set and degrade under distribution shift. Extra-training methods (the various expert pruning lines) require retraining part of the network. ACE sidesteps all three — it judges experts by their own spectral structure and the router's own historical preferences, with signals entirely internal to the model.

So what: the next stop for MoE inference may be "tell the router to be quiet"

The interesting part is not the 7.96% number itself but the direction it points to. Training has already taught the experts what they need to know; inference keeps paying the same compute price for tokens that do not need it. Frameworks like ACE imply that a large share of MoE's efficiency dividend is still unclaimed. Once these ideas land in vLLM, SGLang, or llama.cpp, the bar for running big MoEs on small memory should drop another notch.

Original paper (arXiv:2609.05228): https://arxiv.org/abs/2609.05228