MoE (Mixture-of-Experts) models are sold on a simple pitch: huge total capacity, but each token computes only a small slice. A router picks a handful of experts per token, so total parameters grow without proportional per-token compute. The deployment-side catch is less discussed — compute sparsity does not equal memory friendliness. When the full expert set no longer fits in GPU memory, non-resident experts sit in host memory or slower storage, and every expert the router touches at decode time must be hauled back over the bus. Weight transfer becomes the new bottleneck.

A team from Huawei, Beihang University and partner institutions posted a paper to arXiv on Sep 4 (https://arxiv.org/abs/2609.04895) that picks an unusual place to attack this: instead of guessing "which experts come next" with serving-layer heuristics, make cache management itself a model-side post-training objective.

Two auxiliary routers, native Top-K untouched

The method has two stages. The Temporal Router runs after each MoE layer executes: it predicts same-layer expert reuse for future tokens and decides which experts stay in the GPU cache — retention only, no proactive loading, so it generates no speculative transfers. The Spatio Router exploits causal Transformer execution order: before a target layer is accessed, the hidden state of its causal predecessor is already computed, and that signal refines the cache ahead of the access.

The key constraint: the native MoE router's Top-K expert-selection rule is fully preserved at inference. Auxiliary routers manage residency, never which experts execute. During post-training, a softened Top-B membership surrogate builds a cache-coverage loss jointly optimized with the language-modeling loss, adapting the backbone and auxiliary routers together toward a cache-friendly routing distribution.

The numbers

Evaluated on Qwen3 and GPT-OSS across GSM8K, MATH and CommonsenseQA (five-seed means), the update-only Temporal Router beats the strongest classical replacement policies by 10.46, 10.70 and 33.34 hit-rate points on the three tasks, while per-token decode traffic falls from 1353/1294/1512 MB to 974/906/304 MB. The full Spatio-Temporal Router improves adjusted hit rate by 1.15–18.03 points and cuts traffic by 4.6–53.3% versus ProMoE, the strongest prefetching baseline.

Overhead is remarkably small: the full mode adds 25.2M inference-time parameters on Qwen3 — 0.083% of model size (4.4M, 0.021%, on GPT-OSS) — versus 96.0M for ProMoE. GPT-OSS results are described as "competitive but task-dependent", which the paper does not paper over.

No free lunch

The ablations disclose real costs. Raising the cache-loss weight sw pushes traffic down further but erodes accuracy: at sw=0.1 the Temporal Router keeps baseline accuracy (GSM8K 85.44 unchanged); at sw=2.0 accuracy drops 10.84 points. Expert usage also concentrates toward the head — entropy falls from 3.96 to 3.51 and effective experts shrink from 53.41 to 34.54, which may hurt expert-parallel load balance. The authors state plainly that Load measures simulated decode-stage traffic, not end-to-end latency, and that the method requires full-model post-training rather than being a drop-in serving patch.

So what

The core move here is pulling MoE expert caching from serving-layer heuristics (MoE-Infinity, ProMoE, FineMoE) into model-side post-training. For memory-constrained deployments — think llama.cpp-style projects pushing large MoE models onto consumer hardware — this is a direction worth tracking: cache policy becomes a trainable objective instead of LRU-style guessing. The caveats are equally clear: full-model post-training plus expert concentration side effects mean it is not plug-and-play yet. Would you retrain a model for cache awareness to cut your weight traffic in half?