A 1911 encyclopedia and a child-directed corpus, fed to the same 58M-parameter small model, produce leaderboards that point in opposite directions.

On grammatical acceptability the child-directed corpus wins across the board. On synthetic state tracking, the century-old encyclopedia takes the lead, with gaps reaching more than twenty points. And this is not a tokenizer artifact, a different optimizer, or a quietly longer training run — architecture, recipe, and evaluation protocol are all matched, and the only variable is what the model reads.

A deliberately narrow comparison

The design is unusually disciplined: exactly two corpora. One is the 1911 Encyclopaedia Britannica, 11th edition, scraped from Wikisource (volumes 1–28, index volume excluded, public domain). The other is the BabyLM 2026 strict corpus, a heterogeneous mixture of child-directed speech, dialogue, simple Wikipedia, and transcribed speech. Two source-word budgets: 10 million and 37 million words.

Three model layers. Teachers: at the 10M budget Llama-360M, GPT-2-705M, and GPT-J-705M; at 37M the first two. Students: 58M-parameter Baby-Llama models trained both with knowledge distillation (KD) and without it (no-KD), so corpus effects can be separated from distillation effects. Two training seeds, 2023 and 43. Everything runs on a single NVIDIA A40 in fp16.

Evaluation is entirely zero-shot: BLiMP (67 minimal-pair tasks), BLiMP Supplement, EWoK, Entity Tracking, and COMPS.

One trap deserves to be flagged up front: matching words does not mean matching tokens. Each track trains its own 16,000-token BPE. Britannica comes out at 1.42 tokens per word and BabyLM at 1.95, so at the same 10-million-word budget BabyLM yields roughly 1.37× as many BPE training tokens. The paper later runs a matched-exposure control specifically to close that gap.

Twenty paired comparisons: grammar sweeps, tracking reverses

The paper reports a replicated ranking inversion. All 20 paired BLiMP comparisons favor BabyLM 2026. Of the 20 Entity Tracking comparisons, 18 favor Britannica; only two rows reverse.

For the default Llama+GPT-2 distilled student, averaged over the two seeds: at the 10M budget the deltas are +7.64±0.79 on BLiMP, +10.83±0.86 on Supplement, and −19.63±3.73 on Entity Tracking. At 37M they are +10.55±0.33, +7.08±1.89, and −4.88±0.25.

Absolute scores are more intuitive. For the 10M distilled student: Britannica scores 64.69 on BLiMP and 37.10 on Entity Tracking, BabyLM scores 71.76 and 14.83. At 37M, BabyLM posts 76.27 and 17.10 against Britannica's 65.95 and 21.80.

Entity Tracking is a two-way choice, so chance is 20%. BabyLM's no-KD student manages 23.14 — barely above chance — and distillation drags it down to 14.83, below chance. Britannica's distilled student reaches 37.10, about 11 points above either of its own teachers (26.13 and 25.38).

Distillation is an amplifier, not the cause

The no-KD baseline makes the causal chain plain. At 10M, without distillation Britannica scores 26.11 against BabyLM's 23.14, a gap of roughly 3 points. Add distillation and it becomes 37.10 against 14.83, a gap of roughly 22 points. Distillation did not create the contrast; it amplified it — the paper's own wording is that distillation "amplifies the corpus-level Entity Tracking contrast rather than reducing it."

Could the student simply be inheriting the teacher ensemble? The paper tests that directly: at 10M the Llama+GPT-2 logit mean reaches only 25.15 while the student hits 37.10, a gap of +11.95, so "KD inherits the ensemble" does not hold. But at 37M the ensemble scores 23.85, beating the student's 21.80 — which means the amplification is budget-specific rather than a generic property of Britannica plus distillation.

Two further controls hold up. Capping BabyLM's optimizer steps to Britannica's counts (3,472 steps for Llama, 2,604 for GPT-2, 20,826 per student, matching sequence presentations to within 0.03%) preserves the acceptability advantage in all four model families and preserves the Britannica Entity Tracking advantage in both teachers and the central KD comparison, with only the no-KD row reversing slightly to +1.28. Re-running everything at a 256-token context leaves both the corpus ranking and the student-versus-ensemble pattern intact.

Why this is worth taking seriously

First, a single aggregate score is a bad metric. The paper is blunt about it: reporting only BLiMP would favor BabyLM 2026 throughout, while reporting Entity Tracking would often favor Britannica. Only multi-benchmark evaluation exposes a profile of capabilities rather than one averaged-away number.

Second, corpus is not a quantity problem but a shape problem. Changing what the model reads produces two strong points that point in opposite directions. That is a warning for the era of mixing trillions of tokens: the more uniformly you blend, the blurrier the profile.

Third, at small scale the corpus lever is undervalued. The compute bill here is not intimidating. At the 10M budget a 58M distilled student takes 4–6 hours and a no-KD student only 0.5–0.8 hours. You can entirely afford to run several corpus comparisons.

The boundaries have to be stated

The authors' own limitations are real: with only two corpora (n=2), the link between corpus structure and skill profile is suggestive rather than conclusive. The student side uses a single 58M architecture, a single BabyLlama distillation recipe, and zero-shot evaluation. The full paired sweep uses two seeds, and the magnitude of the Entity Tracking gaps is seed-sensitive — two non-KD rows reverse under seed 43. There is also a point that is easy to miss: Entity Tracking measures state tracking and reference resolution over synthetic discourse, not encyclopedic knowledge, so Britannica winning here does not mean it "knows more." Ethically, the 1911 text carries clearly dated viewpoints, and the paper cautions that the comparison should not be taken as endorsement of the corpus for downstream user-facing models.

So what? When you have one leaderboard and one aggregate score, you do not actually know what shape of model you have raised. To judge whether you picked the right corpus, you need a few more rulers that do not measure the same thing.

Paper: arXiv:2611.07812, published at the Second BabyLM Workshop (EMNLP 2026). Code and configurations: EdoardoVaira/babylm-corpora-skills. ACL Anthology entry: 2026.babylm-main.33.