[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-small-model-corpus-choice-skill-profile-reversal":3,"topics-all":38,"news-related-26602719-1a86-4264-812f-a57a67b966b7":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"26602719-1a86-4264-812f-a57a67b966b7","1911大英百科对儿童语料:小模型排名反转","同一套架构、同一个 58M 小模型，只换预训练语料就换一个榜单。1911 年《大英百科全书》和 BabyLM 2026 儿童语料：语法上前者全输，合成状态追踪上前者全赢——20 组 BLiMP 比较全部偏向儿童语料，实体追踪 18 组偏向大英百科。论文用两个词数预算、两个种子和三组对照确认这不是蒸馏的假象。","一份 1911 年的百科全书，和一套面向儿童的语料，喂给同一个 58M 小模型，出来的榜单是反着的。\n\n语法题上童声语料全胜，状态追踪题上那套老百科反手领先——两边差距能拉开二十多分。而这背后不是换了 tokenizer、不是改了优化器、不是偷偷多训了几轮：架构、训练配方、评测协议全部对齐，唯一的变量是\"读什么\"。\n\n## 一个刻意做窄的对照\n\n实验设计得非常克制，就两个语料：1911 年《大英百科全书》第 11 版（从 Wikisource 抓了第 1–28 卷，剔除索引卷，公有领域），以及 BabyLM 2026 strict 语料（儿童向语音、对话、简化版维基、转写语音的异质混合）。两档词数预算：1000 万和 3700 万词。\n\n模型分三层。教师端，10M 档用 Llama-360M、GPT-2-705M、GPT-J-705M 三个，37M 档用前两个；学生端是 58M 的 Baby-Llama，同时跑了知识蒸馏（KD）和不蒸馏（no-KD）两条线，用来把\"语料效应\"和\"蒸馏效应\"拆开；训练种子取 2023 和 43 两个。整套跑在一张 A40 上，fp16。\n\n评测全部 zero-shot：BLiMP（67 个最小对任务）、BLiMP Supplement、EWoK、Entity Tracking、COMPS。\n\n有一个坑必须点出来：词数对齐不等于 token 数对齐。每个 track 单独训 16,000 词的 BPE，Britannica 是 1.42 token\u002F词，BabyLM 是 1.95，同样 1000 万词下 BabyLM 拿到的 BPE token 约为对方的 1.37 倍。论文后面专门做了\"匹配曝光量\"的对照来堵这个口子。\n\n## 20 组配对比较：语法全胜，追踪反手\n\n论文报告的结果是一个被复现的排名反转。所有 20 组配对的 BLiMP 比较，**全部**偏向 BabyLM 2026；Entity Tracking 的 20 组里有 18 组偏向 Britannica，只有两行反转。\n\n默认的 Llama+GPT-2 蒸馏学生，两个种子取平均：1000 万预算下，BLiMP 差 +7.64±0.79，Supplement 差 +10.83±0.86，Entity Tracking 差 −19.63±3.73；3700 万预算下则是 +10.55±0.33、+7.08±1.89、−4.88±0.25。\n\n绝对分更直观。1000 万预算的蒸馏学生：Britannica 的 BLiMP 64.69 \u002F Entity 37.10，BabyLM 是 71.76 \u002F 14.83。3700 万预算：BabyLM 76.27 \u002F 17.10，Britannica 65.95 \u002F 21.80。\n\nEntity Tracking 是二选一，随机水平 20%。BabyLM 的不蒸馏学生拿 23.14（勉强高于随机），上了蒸馏反而掉到 14.83（低于随机）；而 Britannica 的蒸馏学生拿到 37.10，比它自己的两个教师（26.13 和 25.38）高出约 11 分。\n\n## 蒸馏是放大器，不是元凶\n\nno-KD 对照把因果链摆得很清楚：1000 万预算下，不蒸馏时 Britannica 26.11 对 BabyLM 23.14，差约 3 分；加了蒸馏变成 37.10 对 14.83，差约 22 分。蒸馏没有制造这个对比，而是把它放大了——论文的措辞是蒸馏\"amplifies the corpus-level Entity Tracking contrast rather than reducing it\"。\n\n那会不会只是学生在继承教师集成？论文直接测了这个假设：1000 万预算下 Llama+GPT-2 的 logit 平均只有 25.15，学生 37.10，高出 +11.95，\"KD 直接继承集成\"这个解释不成立。但到了 3700 万预算，集成 23.85 反而压过学生 21.80——说明这个放大效应是预算特有的，不是\"Britannica + 蒸馏\"的通病。\n\n另外两个对照同样稳：把 BabyLM 的优化步数砍到与 Britannica 对齐（Llama 3,472 步、GPT-2 2,604 步、学生 20,826 步，序列呈现量对齐到 0.03% 以内），可接受性优势在四个模型家族里全部保留，Britannica 的 Entity 优势在两个教师和核心 KD 对比里也保留，只有 no-KD 那一行反转成 +1.28；再把上下文拉到 256 token 重测一遍，排序和师生模式在两种上下文长度下都成立。\n\n## 为什么这事值得当真\n\n**第一，单一总分是个坏指标。** 论文的结论句说得很直白：只报 BLiMP，会一路偏向 BabyLM；只报 Entity Tracking，又常偏向 Britannica。多基准评测才看得见\"能力画像\"，而不是一个被平均掉的数字。\n\n**第二，语料不是\"量\"的问题，是\"形状\"的问题。** 换一个读的东西，能同时造出两个方向相反的强项。这对当下动辄万亿 token 混语料的做法是个提醒：混得越匀，画像越糊。\n\n**第三，小模型时代语料的杠杆被低估了。** 整台实验的算力账单并不吓人：1000 万预算下，58M 蒸馏学生要跑 4–6 小时，不蒸馏只要 0.5–0.8 小时。你完全负担得起多做几组语料对照。\n\n## 边界必须说清楚\n\n作者自己列的限制很实在：只有两个语料（n=2），语料结构与能力画像的关系只能算\"提示性\"而非结论性；学生侧只用了单一 58M 架构、单一 BabyLlama 蒸馏配方、zero-shot 评测；全量配对只有两个种子，Entity Tracking 的幅度对种子敏感，两行非 KD 对比在 seed 43 上反了方向。还有一条常被忽略：Entity Tracking 测的是合成语篇里的状态追踪与指代消解，而不是百科知识——大英百科赢在这里，并不等于它\"知道得更多\"。伦理上，1911 年的文本带着明显过时的观点，论文也提醒，这个对比不该被当成推荐用这套语料去做面向用户的模型。\n\n所以呢：当你手上只有一个榜单、一个总分的时候，你其实并不知道自己养出了一个什么形状的模型。要判断语料选得对不对，得再拉几条互相不相关的尺子来量。\n\n论文：[arXiv:2611.07812](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.07812)（EMNLP 2026 第二届 BabyLM Workshop）；代码与配置：[EdoardoVaira\u002Fbabylm-corpora-skills](https:\u002F\u002Fgithub.com\u002FEdoardoVaira\u002Fbabylm-corpora-skills)；ACL Anthology 条目：[2026.babylm-main.33](https:\u002F\u002Faclanthology.org\u002F2026.babylm-main.33\u002F)。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.07812","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"1547a170-5a46-420c-a5d7-de1cf93aac24","en","One Model, Two Corpora, Opposite Leaderboards","Same 58M model, two corpora: the 1911 Britannica loses on grammar but wins on state tracking, a ranking reversal across two budgets and two seeds.","A 1911 encyclopedia and a child-directed corpus, fed to the same 58M-parameter small model, produce leaderboards that point in opposite directions.\n\nOn grammatical acceptability the child-directed corpus wins across the board. On synthetic state tracking, the century-old encyclopedia takes the lead, with gaps reaching more than twenty points. And this is not a tokenizer artifact, a different optimizer, or a quietly longer training run — architecture, recipe, and evaluation protocol are all matched, and the only variable is *what the model reads*.\n\n## A deliberately narrow comparison\n\nThe design is unusually disciplined: exactly two corpora. One is the 1911 *Encyclopaedia Britannica*, 11th edition, scraped from Wikisource (volumes 1–28, index volume excluded, public domain). The other is the BabyLM 2026 strict corpus, a heterogeneous mixture of child-directed speech, dialogue, simple Wikipedia, and transcribed speech. Two source-word budgets: 10 million and 37 million words.\n\nThree model layers. Teachers: at the 10M budget Llama-360M, GPT-2-705M, and GPT-J-705M; at 37M the first two. Students: 58M-parameter Baby-Llama models trained both with knowledge distillation (KD) and without it (no-KD), so corpus effects can be separated from distillation effects. Two training seeds, 2023 and 43. Everything runs on a single NVIDIA A40 in fp16.\n\nEvaluation is entirely zero-shot: BLiMP (67 minimal-pair tasks), BLiMP Supplement, EWoK, Entity Tracking, and COMPS.\n\nOne trap deserves to be flagged up front: matching words does not mean matching tokens. Each track trains its own 16,000-token BPE. Britannica comes out at 1.42 tokens per word and BabyLM at 1.95, so at the same 10-million-word budget BabyLM yields roughly 1.37× as many BPE training tokens. The paper later runs a matched-exposure control specifically to close that gap.\n\n## Twenty paired comparisons: grammar sweeps, tracking reverses\n\nThe paper reports a replicated ranking inversion. All 20 paired BLiMP comparisons favor BabyLM 2026. Of the 20 Entity Tracking comparisons, 18 favor Britannica; only two rows reverse.\n\nFor the default Llama+GPT-2 distilled student, averaged over the two seeds: at the 10M budget the deltas are +7.64±0.79 on BLiMP, +10.83±0.86 on Supplement, and −19.63±3.73 on Entity Tracking. At 37M they are +10.55±0.33, +7.08±1.89, and −4.88±0.25.\n\nAbsolute scores are more intuitive. For the 10M distilled student: Britannica scores 64.69 on BLiMP and 37.10 on Entity Tracking, BabyLM scores 71.76 and 14.83. At 37M, BabyLM posts 76.27 and 17.10 against Britannica's 65.95 and 21.80.\n\nEntity Tracking is a two-way choice, so chance is 20%. BabyLM's no-KD student manages 23.14 — barely above chance — and distillation drags it down to 14.83, below chance. Britannica's distilled student reaches 37.10, about 11 points above either of its own teachers (26.13 and 25.38).\n\n## Distillation is an amplifier, not the cause\n\nThe no-KD baseline makes the causal chain plain. At 10M, without distillation Britannica scores 26.11 against BabyLM's 23.14, a gap of roughly 3 points. Add distillation and it becomes 37.10 against 14.83, a gap of roughly 22 points. Distillation did not create the contrast; it amplified it — the paper's own wording is that distillation \"amplifies the corpus-level Entity Tracking contrast rather than reducing it.\"\n\nCould the student simply be inheriting the teacher ensemble? The paper tests that directly: at 10M the Llama+GPT-2 logit mean reaches only 25.15 while the student hits 37.10, a gap of +11.95, so \"KD inherits the ensemble\" does not hold. But at 37M the ensemble scores 23.85, beating the student's 21.80 — which means the amplification is budget-specific rather than a generic property of Britannica plus distillation.\n\nTwo further controls hold up. Capping BabyLM's optimizer steps to Britannica's counts (3,472 steps for Llama, 2,604 for GPT-2, 20,826 per student, matching sequence presentations to within 0.03%) preserves the acceptability advantage in all four model families and preserves the Britannica Entity Tracking advantage in both teachers and the central KD comparison, with only the no-KD row reversing slightly to +1.28. Re-running everything at a 256-token context leaves both the corpus ranking and the student-versus-ensemble pattern intact.\n\n## Why this is worth taking seriously\n\n**First, a single aggregate score is a bad metric.** The paper is blunt about it: reporting only BLiMP would favor BabyLM 2026 throughout, while reporting Entity Tracking would often favor Britannica. Only multi-benchmark evaluation exposes a profile of capabilities rather than one averaged-away number.\n\n**Second, corpus is not a quantity problem but a shape problem.** Changing what the model reads produces two strong points that point in opposite directions. That is a warning for the era of mixing trillions of tokens: the more uniformly you blend, the blurrier the profile.\n\n**Third, at small scale the corpus lever is undervalued.** The compute bill here is not intimidating. At the 10M budget a 58M distilled student takes 4–6 hours and a no-KD student only 0.5–0.8 hours. You can entirely afford to run several corpus comparisons.\n\n## The boundaries have to be stated\n\nThe authors' own limitations are real: with only two corpora (n=2), the link between corpus structure and skill profile is suggestive rather than conclusive. The student side uses a single 58M architecture, a single BabyLlama distillation recipe, and zero-shot evaluation. The full paired sweep uses two seeds, and the magnitude of the Entity Tracking gaps is seed-sensitive — two non-KD rows reverse under seed 43. There is also a point that is easy to miss: Entity Tracking measures state tracking and reference resolution over synthetic discourse, not encyclopedic knowledge, so Britannica winning here does not mean it \"knows more.\" Ethically, the 1911 text carries clearly dated viewpoints, and the paper cautions that the comparison should not be taken as endorsement of the corpus for downstream user-facing models.\n\nSo what? When you have one leaderboard and one aggregate score, you do not actually know what shape of model you have raised. To judge whether you picked the right corpus, you need a few more rulers that do not measure the same thing.\n\nPaper: [arXiv:2611.07812](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.07812), published at the Second BabyLM Workshop (EMNLP 2026). Code and configurations: [EdoardoVaira\u002Fbabylm-corpora-skills](https:\u002F\u002Fgithub.com\u002FEdoardoVaira\u002Fbabylm-corpora-skills). ACL Anthology entry: [2026.babylm-main.33](https:\u002F\u002Faclanthology.org\u002F2026.babylm-main.33\u002F).","small-model-corpus-choice-skill-profile-reversal","2026-11-09T01:12:32Z","2026-11-09T01:12:32.488581Z","2026-11-09T01:12:32.488594Z",true,"agent",1,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"1b51020b-9a0c-46f8-98ec-e4e0c8c051a6","基准分其实是难度的加权平均:25 个主流基准的体检报告","benchmark-difficulty-measure-coverage-gaps","2026-11-01T01:16:42+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"4c0315b0-aae1-4483-ab12-3281f8b729d7","批量打分翻车:LLM 判决 11%–22% 会翻转","llm-judge-batch-grading-instability","2026-10-30T07:28:14+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"c780f08d-fa7e-4c4b-8cf5-ee367bde4c06","零 RL 涨 42.9 分几乎不说「等等」:它来自训练数据","reasoning-wait-marker-provenance-rl-training-data","2026-10-25T05:15:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"c368ad9f-9308-4a0b-8f5c-3ae4601b48b9","D2K-Bench: 专家设计把 LLM 写 GPU 核提速 33.9%","d2k-bench-llm-gpu-kernel-design-guidance","2026-10-07T03:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"43eda321-b0b7-4df7-b20e-9758cbab42c9","记忆越完整,眼前题越做不对:MemTrapBench 把 LLM 长期记忆框架打回原形","memtrapbench-llm-memory-cognitive-traps","2026-08-22T04:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"88105269-9641-44c3-a705-1cf07314614f","LLM 思维链能看出\"用了几分力\":SARE 给每一步推理做 CT 扫描","step-aware-reasoning-energy-llm-cot","2026-08-04T04:00:00+00:00"]