[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-on-policy-mid-training-distillation-math-collapse":3,"topics-all":38,"news-related-79da91a4-039a-4014-9c4b-44d8dc1b5574":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"79da91a4-039a-4014-9c4b-44d8dc1b5574","同一份语料两种结局:数学分从76.6崩到18.2","一份10亿token数学语料,同一套中期训练配方喂给Llama-3.2-3B,数学分从11.2涨到37.0;喂给Qwen3-1.7B-Base却从76.6崩到18.2,平均分从18.5掉到1.5。论文提出OPMD:用冻结教师在学生自己走出的状态上做逐token反向KL蒸馏,4B教师就能拿到82.8,不加推理开销。","中期预训练（continual pre-training，也叫 mid-training）几乎都是同一套流水线：拿一份领域语料，继续跑下一个 token 预测。但这套默认配方在同一份语料上，能把一个模型推上去，也能把另一个推下悬崖。\n\n11 月 10 日提交的 arXiv 论文（[arXiv:2611.13855](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.13855)）用受控实验把这件事摆到了台面上。作者从 MegaMath-Web-Pro 随机采了 1B token 数学语料，用同一套 NTP 配方，分别喂给 Llama-3.2-3B 和 Qwen3-1.7B-Base，在 MATH-500 和 Minerva 上走出相反的方向。\n\n## 同一份配方，一个涨一个崩\n\nLlama-3.2-3B 的起点是 MATH-500 pass@16 11.2、Minerva 7.3，一轮中期训练后涨到 37.0 和 17.3。这符合直觉：继续喂数学文本，数学能力上升。\n\nQwen3-1.7B-Base 则从 76.6 \u002F 30.5 一路掉到 18.2 \u002F 5.5。MATH-500 的 avg@16 从 18.5 跌到 1.5，Minerva 的 avg@16 从 4.7 跌到 0.4。论文的 NTP checkpoint 表显示这不是中途抖动：第 400 步 30.2，第 800 步 17.8，第 1200 步 17.6，跑满 1902 步 18.2。掉下去没有回来迹象。\n\n作者的解读是：更强的基座有更多已成型的推理行为要保住，反而可能对冲突或低效用的训练信号更脆弱。语料质量不是唯一变量，同一份语料对不同基座的效用可以完全相反。而这正是当前中期训练流水线最盲的地方——主流做法是静态数据筛选，全部发生在优化之前，对模型实际走到哪一步一无所知。\n\n## 换目标函数：在模型自己走过的地方打分\n\n论文的方案叫 OPMD（On-Policy Mid-Training Distillation），把中期训练从离线文本模仿改造成主动查询式知识获取。\n\n三步走。第一步是锚点选择（Anchor Token Identification）：用一个冻结的轻量代理模型 DeepSeek-R1-Distill-Qwen-1.5B 扫文档算逐 token 预测熵，长文档按 4096 窗口、2048 步幅滑动并对重叠区取平均，然后分三组——熵最高的 5%、熵最低的 5%、每个位置以 0.05 概率独立随机预选。第二步是去重与对齐：同一文档按 512-token 非重叠桶分组，每桶只留熵最高的一个候选锚点，三组统一下采样到相同的 435,049 条 anchor prompt，保证比较的只是从哪里开始采样，不是训练了多少。\n\n第三步是核心。学生模型以锚点之前最多 1024 token 的上下文作 prompt，自己 rollout 出最多 512 token 的续写；冻结的教师在这些学生真实访问到的状态上给出逐 token 的 reverse KL 奖励，比较范围限制在学生的 Top-16 候选上。实现跑在 veRL 的 PPO 框架上，学习率 1×10^-6，参考策略 KL 系数 β=0.005，4 到 8 张 A100\u002FA800，一个 epoch 的 AdamW。\n\n## 4B 教师够不够用\n\n效果上，Qwen3-4B-Base 当教师时，MATH-500 pass@16 到 82.8（比基座 +6.2），avg@16 30.5（+12.0）；Minerva pass@16 34.9（+4.4），avg@16 7.0（+2.3）。换成 Qwen3-14B-Base 当教师，MATH-500 是 82.4 \u002F 29.9，Minerva 是 36.0 \u002F 8.0。\n\n两组几乎打平，教师规模翻了三倍多，指标基本没动。论文给了两个可能的解释：一是 1.7B 学生的容量天花板，Top-16 上的软目标差异已经够用；二是状态空间分布对齐，因为 rollout 都由学生自己生成，被访问到的状态锚在学生的分布里，两个教师在这些状态上的相对排序本来就接近。在论文测的两个基准上，教师算力从 14B 降到 4B 没有实质损失。论文也提醒步数不能直接比：4B 教师那组是 4 张卡跑 1699 步，14B 教师那组是 6 张卡跑 1132 步，对应相当的 token 曝光量。\n\n另一个值得记的细节是收敛速度：14B 教师那组在第 100 步，MATH-500 的 avg@16 就从 18.5 涨到 29.4，Minerva 从 4.7 涨到 7.1。\n\n## 锚点选哪没那么关键\n\n消融分高熵、低熵、随机三组，样本预算严格对齐在 435,049。高熵组在两个基准上给出最高的 avg@16 峰值——MATH-500 31.9（第 800 步）、Minerva 8.1（第 800 步）。低熵组明显更差，峰值只有 25.3（第 900 步）和 6.6（第 1699 步），论文的解释是低熵位置多是高度确定的句法结构，蒸馏价值有限。\n\n但随机组的 pass@16 峰值反而更高：MATH-500 83.2（第 800 步），Minerva 40.8（第 400 步）。也就是说，OPMD 这个范式本身对锚点位置相当鲁棒，把静态语料上下文转成主动 rollout 触发才是主要贡献，高熵锚点的优势更多落在解法一致性上。\n\n## 三处必须说清的保留\n\n第一，论文明确把在更多领域上评估、以及用真正均匀随机的最终锚点列为后续工作。而所谓随机组的最终锚点其实并不均匀随机——去重阶段每个 512-token 桶里保留的仍是熵最高的那个，作者在附录里有说明。\n\n第二，NTP 与 OPMD 两条臂的学习率并不相同，NTP 用的是 1×10^-5。解读「同样的语料」时这点不能忽略。论文表格中的 checkpoint 来自同一次训练的连续评估，不是独立重复实验。\n\n还有一个来自独立来源的提醒。OPMD 用的评测脚本 gen_vllm.py 与 grade.py 直接取自清华 THUNLP 团队的 Rethinking On-Policy Distillation 项目（[arXiv:2604.13016](https:\u002F\u002Farxiv.org\u002Fabs\u002F2604.13016)，已被 TMLR 收录）。那篇论文指出 OPD 成功需要两个条件：师生思考模式兼容，且教师能提供学生训练中未见过的真正新能力；他们还发现成功的 OPD 会在学生访问到的状态上、对高概率 token 逐步对齐，一个很小的共享 token 集合就集中了 97% 到 99% 的概率质量。更值得注意的是他们的结论：密集 token 级奖励这个免费午餐是有代价的，OPD 能否扩展到长程蒸馏仍是问号。OPMD 的师生都是 Qwen3 同家族基座，这与「同家族教师在学生视角下可能不可区分」的观察形成张力。\n\n## 所以呢\n\n这套东西的价值不在于又冒出一个蒸馏方法，而在于它把一个一直靠直觉和试错在做的决定——中期训练该用哪种目标函数——变成了可以先小成本验证的工程变量。OPMD 只改训练过程，不加部署时教师、不加额外推理开销。\n\n可执行的版本大概是：中期训练没看到肉眼可见的收益，先别急着加数据，换个目标函数试试，一版对照可能比多爬几百万条语料便宜得多。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.13855","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"5096013b-66d3-4ecf-b75c-1aa5d249af39","en","Same Corpus, Same Recipe: One Model Rises, Another Collapses","One 1B-token math corpus, one recipe: Llama-3.2-3B rose 11.2 to 37.0 on MATH-500, Qwen3-1.7B-Base fell 76.6 to 18.2. OPMD reaches 82.8.","Continual pre-training, also called mid-training, has a default pipeline: take a domain corpus, keep running next-token prediction on it. What this paper shows is that the same default recipe, on the same corpus, can push one model forward and drive another straight off a cliff.\n\nAn arXiv preprint submitted on 10 November 2026 ([arXiv:2611.13855](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.13855)) documents this in a controlled experiment. The authors randomly sampled a 1B-token subset of MegaMath-Web-Pro, applied the same NTP mid-training recipe to two base models — Llama-3.2-3B and Qwen3-1.7B-Base — and got opposite results on MATH-500 and Minerva.\n\n## One recipe, one model rises, one collapses\n\nLlama-3.2-3B started at 11.2 pass@16 on MATH-500 and 7.3 on Minerva, and after one round of mid-training rose to 37.0 and 17.3. That matches intuition: keep feeding mathematical text, mathematical ability goes up.\n\nQwen3-1.7B-Base went the other way, from 76.6 \u002F 30.5 down to 18.2 \u002F 5.5. Its MATH-500 avg@16 fell from 18.5 to 1.5, and its Minerva avg@16 from 4.7 to 0.4. The paper's NTP checkpoint table rules out mid-training noise: step 400 gives 30.2, step 800 gives 17.8, step 1200 gives 17.6, step 1600 gives 20.0, and the final step 1902 gives 18.2. There is no sign of recovery.\n\nThe authors' reading is that a stronger base has more developed reasoning behavior to preserve, and may therefore be more vulnerable to conflicting or low-utility training signals. Corpus quality, in other words, is not the only variable — the same corpus can have sharply different utility for different base models. This is exactly where the current mid-training pipeline is blind: mainstream practice leans on static data curation (heuristic filtering, deduplication, distribution matching, domain reweighting), all of which happens before optimization and none of which knows what states the model actually visits.\n\n## Swapping the objective: score where the model actually goes\n\nThe paper's proposal is called OPMD (On-Policy Mid-Training Distillation), and it recasts mid-training from offline text imitation into active, query-driven knowledge acquisition.\n\nThree steps. First, anchor selection (Anchor Token Identification): a frozen lightweight proxy model, DeepSeek-R1-Distill-Qwen-1.5B, scans each document to compute per-token predictive entropy, with long documents processed in windows of 4096 and stride 2048, averaging entropy across overlapping regions. Three arms are then built — the top 5% highest-entropy positions, the bottom 5% lowest-entropy positions, and a stochastic arm preselecting each position independently with probability 0.05. Second, dedup and alignment: within a document, candidates are partitioned into non-overlapping 512-token buckets and only the highest-entropy candidate in each bucket is kept; the three sets are then uniformly downsampled to an identical 435,049 anchor prompts, so the comparison isolates where rollouts start rather than how much training happened.\n\nThird, the core of it. The student takes the context of up to 1024 tokens preceding the anchor as a prompt and rolls out up to 512 tokens of continuation on its own; the frozen teacher then supplies dense token-level reverse-KL rewards on the states the student actually visits, restricted to the student's Top-16 next-token candidates. The implementation runs on the veRL PPO framework with a learning rate of 1×10^-6, a reference-policy KL coefficient of β=0.005, 4–8 NVIDIA A100\u002FA800 GPUs, and one epoch of AdamW.\n\n## Is a 4B teacher enough\n\nOn results, with Qwen3-4B-Base as teacher, MATH-500 pass@16 reaches 82.8 (+6.2 over base) and avg@16 reaches 30.5 (+12.0); Minerva pass@16 reaches 34.9 (+4.4) and avg@16 7.0 (+2.3). Swap in Qwen3-14B-Base as teacher and MATH-500 gives 82.4 \u002F 29.9 while Minerva gives 36.0 \u002F 8.0.\n\nThe two are close to a tie, even though teacher capacity grew more than threefold. The paper offers two possible explanations: a capacity ceiling for the 1.7B student, since the soft-target signal over the Top-16 set is already sufficient; and state-space distributional alignment, because rollouts are generated by the student itself, so the visited states remain anchored inside the student's distribution and the two teachers assign similar relative log-probabilities to influential candidates there. On the two benchmarks tested, dropping teacher compute from 14B to 4B costs nothing meaningful — which is real money saved for anyone trying to reproduce this. The paper also cautions against comparing step counts directly: the 4B-teacher run took 1699 steps on 4 GPUs, the 14B-teacher run 1132 steps on 6 GPUs, and both correspond to equivalent overall token exposure.\n\nOne more detail worth keeping: convergence speed. In the 14B-teacher run, at step 100 MATH-500 avg@16 had already risen from 18.5 to 29.4, and Minerva from 4.7 to 7.1.\n\n## Where the anchor is picked matters less than you'd think\n\nThe ablation splits into high-entropy, low-entropy and stochastic arms, all held to the same 435,049-sample budget. The high-entropy arm delivers the highest peak avg@16 on both benchmarks — 31.9 on MATH-500 (step 800) and 8.1 on Minerva (step 800). The low-entropy arm is clearly worse, peaking at 25.3 (step 900) on MATH-500 and 6.6 (step 1699) on Minerva; the paper's explanation is that low-entropy positions are mostly highly deterministic structural or syntactic continuations, offering limited exploration value.\n\nBut the stochastic arm posts the higher peak pass@16: 83.2 on MATH-500 (step 800) and 40.8 on Minerva (step 400). So the OPMD paradigm itself is robust to where anchors land; converting static corpus context into active prompt-rollout triggers is the main contribution, and high-entropy selection's advantage shows up more in solution consistency (avg@16) than in peak score.\n\n## Three caveats worth stating plainly\n\nFirst, the paper explicitly lists evaluation across additional domains, and with uniformly random final anchors, as future work. And the \"stochastic\" arm's final anchors are in fact not uniformly random — the dedup step still keeps the highest-entropy candidate in every 512-token bucket, and the authors say so in the appendix.\n\nSecond, the NTP and OPMD arms do not share a learning rate: NTP uses 1×10^-5. That matters when reading the \"same corpus\" framing. And the checkpoint figures in the paper's tables are successive evaluations from a single run, not independent repetitions.\n\nThere is also a reminder from an independent source. The evaluation scripts OPMD uses — gen_vllm.py and grade.py — come straight from the Rethinking On-Policy Distillation project by Tsinghua's THUNLP team ([arXiv:2604.13016](https:\u002F\u002Farxiv.org\u002Fabs\u002F2604.13016), accepted to TMLR). That paper argues OPD succeeds only under two conditions: student and teacher must share compatible thinking patterns, and even with consistent patterns and higher scores the teacher must offer genuinely new capabilities beyond what the student saw during training. It also finds that successful OPD is characterized by progressive alignment on high-probability tokens at student-visited states, with a small shared token set concentrating 97%–99% of the probability mass. More pointed: the authors show that OPD's apparent free lunch of dense token-level reward comes at a cost, raising the question of whether it can scale to long-horizon distillation. OPMD's teacher and student are both same-family Qwen3 bases, which sits in interesting tension with the observation that same-family teachers can be indistinguishable from the student's perspective.\n\n## So what\n\nThe value here isn't that yet another distillation method appeared. It's that a decision previously settled by intuition and trial-and-error — which objective function to use for mid-training — becomes an engineering variable you can validate cheaply. OPMD changes only the training procedure: no teacher at deployment, no extra inference overhead, and it drops into existing CPT pipelines.\n\nThe actionable version sounds like this: if your mid-training shows no visible gain, don't add more data yet — try a different objective function. One controlled comparison may well cost less than scraping a few million more tokens.","on-policy-mid-training-distillation-math-collapse","2026-11-12T11:40:00Z","2026-11-12T11:13:28.799090Z","2026-11-12T11:13:28.799098Z",true,"agent",43,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"029d5b6c-a448-442b-b742-96afeaab330f","PCS 把 LLM 推理能力\"渐进迁移\"到任意语种：5 个语种验证，轻量翻译替代昂贵蒸馏","pcs-llm-progressive-transfer","2026-07-08T14:15:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"90fc8caa-c5f6-45ab-adb8-50f28f43739b","字节 UP：正向 advantage 不裁剪，GRPO\u002FDAPO\u002FGSPO 即插即用","bytedance-seed-up-advantage","2026-07-08T04:21:42+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"e4c13922-29e1-41a9-9470-8dae80f62368","推理模型的「无效思考」:55% 的 CoT 步骤对答案概率毫无影响","epiphenomenal-cot-55pct-useless-thinking","2026-06-14T10:01:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"58645289-9914-4c61-a9bd-3691afa52dff","QK-Restore：给混合注意力LLM装上\"长程记忆保险丝\"，CoT微调后256K检索从65.4%拉回76.4%","qk-restore-long-range-memory-fuse-256k-76pct","2026-06-10T08:20:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"a65df42b-5933-453e-9572-c299187e0551","AI教AI:教师模型写脚手架,RL训练四域涨分最多7.2点","autoscaffold-ai-guided-rl-scaffolding","2026-11-12T15:25:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"78b52428-2dbb-4819-aa2a-eee5bccf7d5a","用大模型筛基准:基准反而变简单了","llm-validation-minimal-pair-benchmarks-easier","2026-11-11T11:20:00+00:00"]