[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-distillation-kl-direction-vs-rollout-policy":3,"topics-all":38,"news-related-638eacdb-6796-40df-bfe1-c48bdf17efe5":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"638eacdb-6796-40df-bfe1-c48bdf17efe5","剑桥拆解蒸馏:KL 方向才是主角,on-policy 不是","剑桥团队把 rollout 策略、KL 方向与学习率拆成独立旋钮,在 Llama 3 与 Qwen2.5 双家族受控实验蒸馏动态:KL 方向决定表现与覆盖,学习率决定遗忘与稀疏更新,rollout 策略未必居中心;forward KL 对数据来源异常鲁棒,on-policy 仅存的泛化优势也难在 RLVR 后保持。","过去两年，\"on-policy\" 几乎成了大模型后训练的标准答案：让模型在自己生成的数据上学习，据说能减少灾难性遗忘、带来更稀疏的更新与更好的泛化。剑桥 van der Schaar 实验室的 Julianna Piskorz 与 Antonin Berthon（共同一作）9 月 28 日提交至 arXiv 的论文，对这套叙事做了受控检验。结论不客气：蒸馏场景下，rollout 策略未必是主角。\n\n## 三个旋钮拆开的对照实验\n\n以往比较监督微调与强化学习的研究往往同时变动多个因素，rollout 策略的真实贡献很难单独剥出来。论文把蒸馏配置拆成三个独立旋钮——rollout 策略、token 级 KL 方向、学习率——在 Llama 3 与 Qwen2.5 两家族上逐一变动，并刻意打破「forward KL 配 off-policy、reverse KL 配 on-policy」的行业惯例。\n\n主实验把 Llama-3.1-8B 教师蒸馏进 Llama-3.2-1B 学生，附录用 Qwen2.5-7B 教师与 Qwen2.5-1.5B 学生复验；任务覆盖科学、医疗（MedReason）、算术（Countdown）三个推理域；教师基准一并公开：Llama 教师在 MedReason 上准确率 85.5%，Qwen 教师在 Countdown-3 上达到 95.5%。\n\n## 真正的决定因素换了人\n\n核心结论一句话：token 级 KL 方向更清楚地决定任务表现与输出覆盖，学习率决定灾难性遗忘与更新稀疏性，rollout 策略不必然居中心。拆开看：\n\n- **forward KL 对 rollout 策略异常鲁棒**——无论数据由谁生成，表现都稳定且强。论文用梯度分析给出解释（定理 C.1）：forward KL 梯度范数平方的期望存在与 rollout 策略无关的上界。\n- **reverse KL 敏感得多**，明显偏好学生自己生成的 rollout。\n- 作者明确写道：**没有证据表明 on-policy rollout 本身能保留既有能力或诱导更稀疏的更新**。\n\n## on-policy 什么时候还有用\n\n论文没有一棍子打死。Countdown 的更难变体上，on-policy 数据在两种 KL 方向下都带来一致的泛化提升——只是该优势在后续 RLVR 训练后不再可靠保持。另一个发现：reverse KL 搭配 on-policy 能减少「教师风格」的非任务迁移，论文专门设计了被诱导用西班牙语推理的教师实验验证。\n\n## 所以呢\n\n工程含义：蒸馏预算有限时，优先投在选对 KL 目标与调好学习率，而非默认铺开昂贵的 on-policy 管线——它意味着教师模型全程在线推理。论文原文正是「鼓励实践者把 on-policy 蒸馏的额外成本与更便宜的替代方案放在同一架天平上称量」。它把问题从「要不要 on-policy」改写成「在什么目标、设定、超参区间里，on-policy 才值回票价」。\n\n论文：[On-Policy or Off-Policy Learning?](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.35259)。发布两天，已在 Hugging Face Daily Papers 收获 80 个赞。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.35259","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"15523c78-84d3-4431-9782-2f271ce3dff6","推理",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"b8b8c9a6-a8e8-4217-9824-daa280188891","en","KL Direction, Not On-Policy, Decides Distillation Outcomes","In controlled Cambridge distillation tests, KL direction drives performance, learning rate drives forgetting, and on-policy rollouts aren't central.","For the past two years, \"on-policy\" has been the default answer in LLM post-training: train on the model's own rollouts, and you supposedly get less catastrophic forgetting, sparser parameter updates, and better generalisation. A controlled study from the van der Schaar Lab at the University of Cambridge — submitted to arXiv on September 28 by co-first authors Julianna Piskorz and Antonin Berthon — puts that narrative to a proper test, and the verdict is unkind: in distillation settings, rollout policy is not necessarily the protagonist.\n\n## A controlled experiment with three separate knobs\n\nPrevious comparisons between supervised fine-tuning and reinforcement learning varied many factors at once, making it hard to isolate what rollout policy actually contributes. This paper decomposes the distillation configuration into three independent knobs — rollout policy (student-generated or teacher-generated), token-level KL direction (forward or reverse), and learning rate — and varies each one separately across the Llama 3 and Qwen2.5 model families, deliberately breaking the conventional pairing of forward KL with off-policy learning and reverse KL with on-policy learning.\n\nThe main experiments distil Llama-3.1-8B teachers into Llama-3.2-1B students, with an appendix replicating everything using Qwen2.5-7B-Instruct teachers and Qwen2.5-1.5B-Instruct students. Tasks span three reasoning domains: science, medical (MedReason), and arithmetic (Countdown). Teacher baselines are disclosed too: the Llama-3.1-8B teacher reaches 85.5% accuracy on MedReason, while the Qwen2.5-7B teacher hits 95.5% on Countdown-3.\n\n## The real deciding factors are different ones\n\nThe core finding fits in one sentence: token-level KL direction more clearly determines task performance and output coverage, learning rate governs catastrophic forgetting and update sparsity, and rollout policy does not necessarily play a central role. Broken down:\n\n- **Forward KL is remarkably robust to rollout policy** — performance stays stable and strong no matter who generates the data. The paper backs this with gradient analysis (Theorem C.1): the expected squared norm of the forward-KL gradient has an upper bound independent of the rollout policy.\n- **Reverse KL is substantially more sensitive**, and clearly favours student-generated rollouts.\n- The authors state plainly: **there is no evidence that on-policy rollouts intrinsically preserve prior capabilities or induce sparser updates**.\n\n## When on-policy still helps\n\nThe paper does not dismiss on-policy entirely. On harder variants of the Countdown arithmetic task, on-policy data consistently improves generalisation under both KL directions — although this advantage does not reliably persist after subsequent RLVR training. Another fine-grained finding: reverse KL combined with on-policy rollouts reduces incidental teacher-style transfer, which the authors verified with a dedicated experiment using a teacher induced to reason in Spanish.\n\n## So what\n\nThe direct implication for engineering teams: if your distillation budget is limited, spend it first on choosing the right KL objective and tuning the learning rate, not on rolling out an expensive on-policy pipeline by default — on-policy means the teacher model runs inference throughout training. The paper's own words encourage practitioners \"to weight the additional cost of on-policy distillation over cheaper alternatives\" on the same scale. It reframes the question from \"on-policy or not\" to \"under what objective, evaluation setting, and hyperparameter regime does on-policy actually pay for itself\".\n\nPaper: [On-Policy or Off-Policy Learning?](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.35259). Two days after release, it has gathered 80 upvotes on Hugging Face Daily Papers.","distillation-kl-direction-vs-rollout-policy","2026-10-04T19:15:00Z","2026-10-04T19:13:05.743470Z","2026-10-04T19:13:05.743477Z",true,"agent",350,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"28c41f06-d20f-481c-b133-cd109af3aed1","答对之后停不下来:微软团队揪出在线蒸馏的 EOS 错配元凶","eos-mismatch-opd-length-inflation","2026-09-18T21:09:06+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"9825e20d-c9eb-4300-99b5-12eb7d0e755d","自信的错误教师最危险:TGOPD 给在线蒸馏装提示级门控,教师 GPU 利用率 9.8% 升至 78.9%","tgopd-teacher-gated-on-policy-distillation","2026-09-08T23:10:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"58ed753e-ad6d-4aac-95f4-36bf217e169c","把 10 万条人类视频变成机器人教材:RoboTok 检索 mAP 提升约 50 倍,hard 任务 79.3% 对 19.5%","robotok-retrieval-benchmark-reread","2026-09-06T21:11:25+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"005557c5-8a3c-4d34-89bc-35d5351c4570","蒸馏只需要一条训练样本?清华实测:单条query覆盖71.5%训练状态,16条追平17k全量","one-shot-opd-single-query-distillation","2026-09-05T21:07:11+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"7623f190-7071-4811-a6f1-32462a99b8d3","经验会过期:阿里云论文让自主后训练的有害授权率从 62.5% 降到 25%","bcit-conditional-experience-transfer-post-training","2026-09-05T17:11:11+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"4a89fe5a-8703-49e5-b083-079cbda0fa2a","蒸馏也有副作用:中间训练期上KD,推理上涨、事实记忆反而变慢","switch-distillation-midtraining-kd","2026-09-02T17:10:00+00:00"]