Mixture-of-Experts (MoE) models carry a popular intuition: each expert handles its own turf — a code expert for code, a medical expert for medicine. A rival line of research keeps pushing back, arguing that routing decisions are driven mostly by token identity and hidden-state geometry, not domain expertise. A new 55-page arXiv paper offers a verdict that lets both sides keep their dignity: they were just looking at the wrong layer. The real domain organization lives not at the expert level but at the neuron level.
Neurons Are the Unit of Specialization
The authors propose Routing-Aware Magnitude Profiling (RAMP), which weighs each neuron's contribution magnitude by its expert's routing coefficient. Calibrating on nine groups of mixed-domain text spanning chat, mathematics, medicine, Chinese content and code, across six models — Qwen3-30B-A3B, Qwen3.5-35B-A3B, Qwen3.6-35B-A3B, GLM-4.7-Flash, Qwen1.5-MoE-A2.7B-Chat and OLMoE-1B-7B-0125-Instruct — the result is remarkably consistent: at a 6.25% selection rate, neuron sets from the same domain overlap 58-74% across samples, while cross-domain overlap is only 15-24%. The domain signal clearly exists — at the neuron level.
The distribution pattern is even more telling: 62.64-82.50% of domain-exclusive neurons reside in experts where they are disproportionately represented, and this enrichment correlates positively with expert-level routing preferences (median Spearman correlations of 0.42-0.71). This is exactly why one camp measures domain preferences while the other finds no functional boundaries: domain neurons are scattered across multiple experts, and expert-level statistics average the signal away.
Training Dynamics: Calm on Top, Churning Below
The paper also traces neuron importance across training — following the full OLMoE training trajectory, plus staged combinations of OpenMoE and Instella. Between adjacent pre-training checkpoints, expert-level importance rankings look stable, yet neuron-importance rank correlations sit at only 0.842-0.939: the layer below keeps reorganizing. During post-training, correlations rise to 0.997-0.998, essentially frozen. Watch only the expert level and you badly underestimate how much reorganization training actually does.
The Practical Ledger for Pruning
The analysis tool becomes a compression tool, RAMPr: instead of deleting whole experts, it selects neurons to retain across expert boundaries, in a cumulative-importance variant (RAMPr-L) and a domain-balanced variant (RAMPr-D). The ledger: at 75% neuron retention, six models preserve 91-98% of original full performance on average; at 50% retention, resident GPU memory drops 36-43% while batched prefill throughput improves 17-39%. On GLM-4.7-Flash, RAMPr-D beats the expert-pruning baseline REAP by 2.91 percentage points on an eleven-task macro average.
So What
This paper pays off for two audiences. For deployment folks: GPU memory is hard currency, and dropping half the neurons costs only single-digit percentages of score — private deployments and resource-constrained settings benefit immediately. For interpretability researchers: it turns the expert-specialization fight into a measurable question — pick the wrong granularity and the conclusions will simply clash. Worth noting: this kind of full-trajectory analysis is only possible on fully open MoE families like OLMoE, OpenMoE and Instella; the paper itself is evidence of the research dividend of open weights. The next question almost asks itself: if neurons are the functional unit, was partitioning capacity by experts the wrong granularity from day one?