Mixture-of-Experts training has a bottleneck that has nothing to do with parameter count. It sits in the backward pass.

In a typical expert-parallel setup, each token is routed to a small set of experts, and those token-expert pairs travel across GPUs through all-to-all communication. The catch is that routing is data-dependent: a few popular experts suddenly receive a flood of tokens while others sit idle. As the paper puts it, an MoE layer and a collective communication both finish at the pace of their slowest participant, so the busiest GPU becomes a straggler and the rest of the cluster waits with it.

Most existing systems work by redistribution: move experts, replicate them, adapt placement dynamically. The paper lists FasterMoE, FlexMoE, SmartMoE, SYMI, LAER-MoE, and UltraEP as representative. The new work argues that path has hit its ceiling. Extra expert copies consume the very GPU memory they were meant to relieve, competing directly with the token-related buffers they were supposed to protect. Under a fixed memory budget, redistributing a workload without shrinking it can leave total demand above what the cluster can actually serve.

The system is called LRMoE (arXiv:2610.37720), from a team at JD.com in Beijing.

Backward demand is not a constant

The core move is to treat backward demand as a scheduling variable rather than a fixed consequence of forward routing. Which pieces of backward work are worth doing becomes a decision the system actively makes, not an outcome the router hands over.

The mechanism is split across two phases. The forward pass still runs the complete routing and computation, preserving all routed work so forward accuracy is unaffected. But before the expert forward begins, the controller observes the routed demand per expert and fixes a backward retention quota for each one. Only when gradients flow back does the system decide which specific token-expert pairs fill those quotas.

LRM: a decomposable contribution score

The paper defines a Load Recalibration Metric. For a routed pair u = (t, e), its contribution to the weight gradient of a protected expert projection is an outer product G_u = (α_u g_t) φ_u^T. Because that product is rank one, measuring the squared Frobenius norm of the perturbation caused by dropping the pair factorizes into three terms:

R_u = α_u² · ||g_t||²₂ · ||φ_u||²₂

  • Routing strength α_u²: how strongly the token was assigned to that expert during forward.
  • Backward signal ||g_t||²₂: the magnitude of the gradient propagated from later computation.
  • Expert-state magnitude ||φ_u||²₂: the magnitude of the saved projection input.

The third term is the one that matters most and is easiest to miss. Because the contribution is an outer product, two routed pairs can arrive with nearly identical upstream gradient norms while carrying very different routing weights or saved activations. A gradient-only rule scores them as interchangeable; their actual effect on the expert weight update is not.

The part that is easy to get wrong

Selection alone does not buy anything. The paper spends a section on exactly this trap. If dropped rows are merely zero-masked while the tensor shape stays fixed, the GEMM dimensions and the memory footprint barely move. Nothing is saved.

What works is physical compaction. The retained gradient rows and their matching saved activations are repacked into contiguous expert-specific buffers, and the actual retained row counts drive variable-size grouped GEMMs with no padding. Each expert then processes only the rows it kept, instead of computing over zero-filled rows.

The figure in the paper makes the consequence concrete: expert loads that were [4, 2, 3] become [3, 1, 2]. An expert whose retained workload now fits on one GPU no longer needs a physical replica. The set of logical experts is untouched; what shrinks is the redundant replica count, along with its parameter memory, temporary buffers, and communication.

What the measurements show

The setup is eight NVIDIA Hopper GPUs, EP8, PyTorch 2.11.0, CUDA 13.0, BF16 precision. The models are Qwen3-30B-A3B and GLM4.5-106B-A12B, trained on a locally preprocessed Alpaca corpus with sequence length 8,192 and a per-GPU microbatch of four. Every compared method runs identical controlled tensors under the same 80 GiB per-GPU memory budget, with 50 warm-up iterations followed by 1,000 measured ones. The baselines are Megatron-LM's fixed-ownership strategy, native UltraEP, and an independent implementation of SYMI (NSDI 2026).

On convergence, the numbers are close to lossless. Averaging the final 1,000 steps of a 10,000-step run on Qwen3-30B-A3B, mean perplexity is 1.015190 with no dropping (C0), 1.015156 at C10, and 1.015141 at C20. Pushing to C30 lands at 1.016270, roughly 0.106% above the C0 baseline. The ablation is where the design earns its keep: LRM records a mean ΔPPL of 0.012474 on Qwen3-30B-A3B and 0.003464 on GLM4.5-106B-A12B, against 0.097654 / 0.011987 for squared gradient norm, 0.107485 / 0.016662 for absolute gradient magnitude, and 0.775797 / 0.079282 for random dropping. Random selection is more than an order of magnitude worse.

On speed, the paper measures cumulative MoE-layer training time to reach a fixed loss target rather than time per step, which is the harder and more honest framing. Against SYMI, the reduction runs 21.2% to 44.8%, a band that spans both evaluated models: on Qwen3-30B-A3B it is 42.5% and 43.2%, while the low end of the range comes from GLM4.5-106B-A12B. For Qwen3-30B-A3B reaching a loss of 0.02, SYMI needs 64.6 minutes over 8,440 steps while LRMoE needs 36.7 minutes over 8,693 steps. More steps, less wall-clock time, and the execution savings more than cover the additional optimization steps. This metric covers execution inside the MoE layers only, excluding overhead outside them, so it is not the same as end-to-end training time.

Memory tells a similar story. The three-component subtotal of base expert placement, additional replica storage, and token buffers measures 12.88 GiB for Megatron-LM, 8.03 GiB for UltraEP, 12.88 GiB for SYMI, and 5.44 GiB for LRMoE, 57.8% below the fixed-ownership baseline. The token-buffer component alone drops from 12.46 GiB under fixed ownership to 4.97 GiB. The backward execution subtotal falls from 29.64 ms to 12.69 ms per MoE layer.

Under 20x batch-size pressure, LRMoE preserves 100% effective throughput while UltraEP manages 58.4%, SYMI 44.9%, and Megatron-LM 40.0%.

Three things worth flagging

First, the entire gain depends on the repacking step. The savings do not come from training on fewer tokens. They come from not shuttling the discarded gradients around. Selection without compaction would be close to worthless.

Second, the paper is honest about its own limits. The retained update is approximate by construction, and the appendix works through how the selection guarantee relates to aggregate gradient error and to subsequent training behavior. The guarantee covers an additive local calibration objective, not end-to-end convergence equivalence. With 20% dropping essentially free and 30% showing a visible gap, the operating point still looks conservative.

Third, this is an engineering result, but it relocates the blame. MoE training waste has long been attributed to uneven routing. This work points at how much memory you reserved for the work in the first place. Once memory is the binding constraint, cutting the least valuable work first beats trying to spread the remaining work more evenly.

The next battlefield in MoE training may not be about balancing load more evenly. It may be about deciding which load was never worth computing.