Large-scale model training is running into a data wall: the supply of human-written text is finite, and repeating training data has shifted from a stopgap to standard industry practice. Meanwhile, frontier models have almost uniformly moved to Mixture-of-Experts (MoE) architectures, trading sparse activation for compute efficiency. What happens when these two trends collide? Until now, almost nobody had studied it systematically. On September 10, a Stanford–University of Washington team posted arXiv:2609.11917, "Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data," authored by a group including Jure Leskovec, Percy Liang, and Luke Zettlemoyer. The verdict is uncomfortable: MoEs are more fragile under repeated data than dense models.
Degradation starts at 4x repetition
The team ran compute-matched grid experiments: 80M, 200M, and 1B active parameters (up to 8.5B total), expert counts from 8 to 128, granularities from 1/2 to 1/32, repetition rates swept from 1 to 1024, across the four domains of the OLMoE data mix — web crawl (DCLM), code (StarCoder), scientific text (peS2o), and Wikipedia.
The core numbers: 80M dense models can repeat data 8x with minimal degradation, while same-scale MoEs start suffering at 4x and are overtaken by dense models at 32x, wiping out the advantage they hold on all-unique data. Crucially, degradation tracks total parameters, not active ones: the sparser the model, the harder it falls. The pattern holds across all four data domains and mixture ratios, and quality filtering barely changes it.
Mechanism: early router ossification, expert over-specialization
Why? The mechanistic analysis offers two lines of evidence. First, MoE routers "ossify" early in training: routing stability reaches 60% by step 400 (about 10% of training) and exceeds 95% by the end — so each expert keeps updating on the same small shard of tokens for most of training, and once data is repeated, it sees that same shard even more often. Second, expert-knockout experiments: raising repetition from 1 to 32 increases knockout cost 2.3x for the 80M, 128-expert, 1/4-granularity configuration, and expert specialization correlates clearly with overfitting.
Remedies exist, but none are complete
The regularization experiments offer a lifeline: methods that drop parameter outputs — dropout, FFN output masking, expert dropout, expert output masking — all work, and with strong masking-based regularization MoEs still beat dense models at more than 64x repetition. Weight decay, gradient clipping, and router jitter show no measurable effect. The paper is also honest that no method fully matches all-unique data. One practical finding: mixing a repeated domain into enough non-repeated data lets the unique portion act as a regularizer.
So what
The real contribution is not the headline "MoEs overfit" — it is turning an industry-default architecture choice into a conditional statement: the dividend of sparsity depends on your unique-token budget. With abundant data, MoE is a free lunch; when data runs dry, it is more fragile than dense models. For teams planning the next base model, counting how much truly unique data is left before deciding how much sparsity to build in may be worth more than a few thousand extra GPUs. Paper