MoE architectures carry a rarely-examined default assumption: every expert belongs to the layer it sits in. Each router picks experts from within its own layer, and the expert pools of different layers stay mutually independent. A paper from Tsinghua University and Shanghai AI Laboratory takes that assumption apart — experts in different layers are topologically independent, yet the content they store turns out to be highly similar.

First, a ruler that compares across architectures

The paper builds a measurement instrument before proposing any optimization. The authors use linear-CKA to quantify pairwise expert similarity: feed every expert the same probe tokens, subtract the sample mean from its response matrix, compute the covariance, then take a cosine similarity in that covariance feature space to obtain a Gram matrix K over the experts. K is positive semi-definite, and its eigenvalue spectrum distributes the similarity energy across principal directions. The paper's headline metric is the Gini coefficient rho of that spectrum: rho=0 means energy is spread evenly and experts are fully complementary, while rho approaching 1 means energy concentrates on a few directions, i.e. heavy redundancy. One elegant property: for a uniform rank-k spectrum the Gini coefficient equals exactly 1-k/E, so rho is literally the fraction of expert directions that have collapsed. Because it is a ratio, it can be compared across expert counts, depths, and training budgets.

The first thing the ruler shows: the higher the proportion of Meta Experts, the lower both loss and redundancy. Counterintuitively, giving Local Experts a higher activation share does reduce redundancy, but at the cost of worse loss. The paper's explanation is that a Local Expert is activated in only one layer, so it is harder to train to sufficiency — under-training is what produces the redundancy.

How much is actually saved

The scaling experiments run a four-rung depth ladder of Qwen3.5-style text-only MoE models at depths L in {12, 16, 20, 24}, trained from scratch on H200 GPUs. Every layer carries 128 routed experts plus one gated shared expert, top-8 routing, attention alternating in a fixed period-4 pattern of three gated-delta-net (GDN) linear-attention layers and one full-attention layer, hidden size growing from 768 to 1536, optimized with Muon under a warmup-stable-decay schedule. Token budgets run from 5.31B to 34.92B and training compute spans 7.135e18 to 2.413e20 FLOPs.

With the total expert count and total activation count locked — so FLOPs are essentially matched — the shared-pool version matches the same cross-entropy using only 70–80% of the original experts, or 75–90% of the parameters. Combining the best configuration from each ablation, the savings come to roughly 20–30% of experts and 15–25% of parameters relative to Vanilla, and the advantage grows as models get deeper and larger.

The topology is not an afterthought

The paper also sweeps the shape of the shared pool. Confining some layers to Local Experts only, or grouping Meta Experts into 2, 4, or 8 sub-pools accessed in a loop, generally degrades loss; mild interventions, by contrast, slightly help. The combination that performs best at every compute budget is: Local Experts reserved for the first two and last two layers, plus Meta Experts partitioned into two interleaved sub-pools. The authors infer from this that the shallowest and deepest layers serve the most differentiated functions and are essentially free of mutual redundancy — which explains why boundary layers must keep dedicated experts. A pure-Meta configuration achieves slightly better loss but is training-unstable, so the adopted baseline is the 75% ratio, called MR75.

The downstream numbers are not tidy

This is the section worth reading. The paper trains a 6B-A1.6B model on 200B tokens and compares five variants across MMLU, C-Eval, CMMLU, BBH, TriviaQA, and OpenBookQA. MR75+Loop2 reaches 0.4175 on MMLU, 0.3640 on OpenBookQA, and 0.4487 on C-Eval, all above Vanilla's 0.4087, 0.3500, and 0.4354. But on BBH, Vanilla's 0.2972 is higher than MR75's 0.2912, and it takes MR75+F2L2 to recover to 0.3061; the best TriviaQA result, 0.2042, comes from MR75+F2L2+Loop2 against Vanilla's 0.1924. The paper is notably restrained in its own framing: given the limited training budget, different variants show advantages on different datasets and larger-scale experiments are needed, yet the variants with shared experts consistently outperform Vanilla overall.

Sharing turns the residual path bidirectional

The theoretical section may be the most valuable part of the paper. In a Vanilla layer-wise MoE, content written by layer l's experts can only be read by deeper layers, and because the read operator differs from the write operator, it arrives as a frozen blended component. The backward gradient highway can carry deep-layer signal to shallow parameters, but that is parameter-level rather than activation-level, and the deep signal gets diluted by the training signals of all intermediate layers. With sharing, every layer's gradient writes into one and the same parameter block theta, so after training theta holds a superposition of all layers' writes — a shallow layer can now read content that deeper layers wrote. The residual path becomes bidirectional instead of one-way.

The authors tighten this claim in the appendix themselves: the path through the shared pool has O(1) length independent of layer distance, which does beat both pre-existing layer-wise routes that pass through |m-l| Jacobians. But the causal direction of a single forward pass is unchanged — what a shallow layer accesses is content written into the shared pool during training, not the activations of deeper layers within the same pass. The analysis characterizes access and path length, and does not guarantee against vanishing signals.

Already running at 35B

The paper's submitter states on the HF paper page that the method has been scaled up in Intern-S2-Mobius. That project's GitHub README describes: a 35B foundation model on the Mobius-v0 architecture, organizing knowledge into a globally shared Memory while multiple Reasoners iteratively query and refine hidden states, built around Backward Residual Connection and Dynamic Latent Reasoning, continually pre-trained from Qwen3.5-35B and then post-trained with SFT and RL, with nearly 4x end-to-end inference speedup claimed in the technical report; the HF model card lists 36B parameters. Attribution matters here: those numbers all come from the project's own README and model card, so they are the vendor's own figures, not a third-party reproduction.

So what

What is genuinely interesting is not the parameter saving itself — a 20–30% expert reduction would rank nowhere near the top of any MoE efficiency discussion. What matters is that the work quietly overturns one of MoE's rarely questioned premises: experts do not have to be layer-private. If expert pools can be shared across depth, redundancy stops being an unavoidable cost of depth and becomes capacity that can be reclaimed.

Still, the conclusion currently rests on limited ground. The bulk of the experiments is a small from-scratch ladder, the downstream evaluation at 6B-A1.6B is far from tidy, and BBH even shows a regression. The real test is whether the shared pool still holds at production scale, where the redundancy structure between experts may look entirely different and where communication and memory overhead live in a different order of magnitude. The authors list "extending to larger models and broader architectures" as future work — and Intern-S2-Mobius at 35B is precisely the starting point that still has to be validated.

Paper: arXiv:2610.27598