[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-on-policy-distillation-teacher-free-opsa":3,"news-related-86c380ed-bdb5-47d0-bf9a-3c55f8573d61":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"86c380ed-bdb5-47d0-bf9a-3c55f8573d61","on-policy 蒸馏真的在蒸馏吗?普渡论文:固定负优势就能追平教师","普渡团队量化发现 on-policy 蒸馏中教师打分噪声随教师规模增大,而学生对此无感;换固定负优势即可追平,说明收益主要来自压制低概率 token。由此提出无监督的 OPSA,让 Qwen3-1.7B 在 AIME24 的 Avg@32 提升 35.41 分,反超 OPD 16.77 分。","在推理模型的训练菜单里，on-policy 蒸馏（OPD）近一年被当成 RLVR 的替代品：RLVR 只在结果层面给稀疏的奖励信号，OPD 则由教师模型对学生自己生成的每一步打分，提供 token 级的稠密监督。听起来更精细，也更贵——你得在训练回路里一直挂着那个大教师。\n\n## 教师的信号，可能比想象的更脏\n\n普渡大学 Yi Ding 与 Ruqi Zhang 8 月 31 日挂在 arXiv 的论文（2608.31046）做的第一件事，就是量化这份监督的质量。结论有点尴尬：教师打分里存在大量噪声，而且教师规模越大，噪声占比越高。原因在结构上就埋好了——教师打分的是学生生成的轨迹，这些轨迹对教师而言天然是 off-policy 的，它在自己不熟悉的分布上打分，可靠性自然存疑。\n\n但更反直觉的是第二层发现：学生对这些噪声几乎无感。把噪声监督保留或剔除，学生最终收敛到的性能差不多。\n\n## 那提升到底从哪来？\n\n作者继续拆，发现学习信号集中在低对数概率的 token 上；更狠的一刀是，把教师提供的优势换成一个固定的负优势值，性能居然追平。这指向一个不太舒服的解释：OPD 的收益，大部分来自「压制低概率 token」这个动作本身——而这个动作，根本不需要教师。\n\n## OPSA：把教师请出训练回路\n\n顺着这个结论，论文提出了 On-Policy Self-Adaptation（OPSA）：不用任何监督信号，用熵自适应的负优势，在高熵位置给更强的学习信号，压掉尾部 token，把概率质量摊回头部 token 上。论文报告的数据：以 Qwen3-1.7B 为基座，OPSA 在 AIME24 上把 Avg@32 提升 35.41 分（相对提升 263%），在全部三个基准上 Pass@32 翻倍以上，并比 OPD 本身在 AIME24 的 Avg@32 高出 16.77 分。论文还声明结论跨模型家族和任务成立，全篇 20 页、12 张图。\n\n这篇论文目前排在 Hugging Face Daily Papers 榜首（截稿时 86 个赞），社区显然被「教师可能白请了」这个论点戳到了。\n\n## 所以呢\n\n对做训练的人来说，可操作的启示不是「马上扔掉教师」，而是先做消融：如果你的 OPD 收益大头来自压制效应，那教师推理预算就是在为一个人的工作付两个人的钱。对行业的隐喻则更冷一点——我们为「蒸馏」这个词支付的品牌溢价，可能远大于它实际传递的知识量。当然，这是单篇论文的结论，AIME24 这类数学基准的表现也不等于通用能力；但「固定负优势追平教师」这个实验本身，值得每个蒸馏叙事的相信者亲手复现一次。\n\n论文：https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.31046","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.31046","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"c187600e-804c-4697-b828-1e4330e0eb10","qwen",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"763cb9d7-9c31-49c5-bbf1-285f23482c04","en","On-Policy Distillation May Not Need a Teacher: Purdue's OPSA Beats It on AIME24","Purdue: on-policy distillation suppresses low-probability tokens; its teacher-free OPSA lifts Qwen3-1.7B AIME24 Avg@32 by 35.41 points, 16.77 above OPD.","On the training menu for reasoning models, on-policy distillation (OPD) has spent the past year cast as an alternative to RLVR: where RLVR hands out sparse outcome-level rewards, OPD keeps a teacher model scoring every step the student generates — dense, token-level supervision. It sounds more precise. It is also more expensive, because the big teacher stays in the training loop the whole time.\n\n## The teacher's signal is dirtier than you think\n\nThe first thing Yi Ding and Ruqi Zhang of Purdue University do in their August 31 arXiv paper (2608.31046) is quantify the quality of that supervision. The finding is awkward: teacher scores carry substantial noise, and the larger the teacher, the higher the noise prevalence. The cause is structural — the teacher is scoring student-generated trajectories that are inherently off-policy for it, grading a distribution it does not normally work in.\n\nThen comes the counter-intuitive layer: the student barely notices. Keep the noisy supervision or strip it out, and the student converges to comparable performance either way.\n\n## So where do the gains come from?\n\nThe authors kept digging and found learning signals concentrating on low-log-probability tokens. The sharper cut: swap the teacher-provided advantage for a single fixed negative advantage, and performance still matches. Which points to an uncomfortable explanation — most of OPD's benefit comes from the act of suppressing low-probability tokens itself. And that act needs no teacher at all.\n\n## OPSA: showing the teacher the door\n\nFollowing that conclusion, the paper proposes On-Policy Self-Adaptation (OPSA): no supervision signal, entropy-adaptive negative advantages that assign stronger learning signals at high-entropy positions, suppress tail tokens, and redistribute probability mass back across head tokens. Numbers as reported by the paper: on a Qwen3-1.7B base, OPSA lifts Avg@32 on AIME24 by 35.41 points (a 263% relative gain), more than doubles Pass@32 across all three benchmarks, and beats OPD itself by 16.77 Avg@32 points on AIME24. The paper also claims the findings generalize across model families and tasks, across 20 pages and 12 figures.\n\nThe paper currently tops the Hugging Face Daily Papers board (86 upvotes at the time of writing) — the community clearly felt the \"paying a teacher for nothing\" sting.\n\n## So what\n\nFor training practitioners, the actionable takeaway is not \"fire the teacher immediately\" — it is \"run the ablation first.\" If most of your OPD gain comes from the suppression effect, your teacher-inference budget is paying two salaries for one person's job. The industry-level metaphor is colder: the brand premium attached to the word \"distillation\" may far exceed the knowledge actually transferred. Caveats apply — this is a single paper, and AIME24-style math benchmarks do not equal general capability. But the fixed-negative-advantage experiment is one every believer in the distillation narrative should try replicating by hand.\n\nPaper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.31046","on-policy-distillation-teacher-free-opsa","2026-09-01T15:05:00Z","2026-09-01T15:08:42.132186Z","2026-09-01T15:08:42.132195Z",true,"agent",72,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"4a89fe5a-8703-49e5-b083-079cbda0fa2a","蒸馏也有副作用:中间训练期上KD,推理上涨、事实记忆反而变慢","switch-distillation-midtraining-kd","2026-09-02T17:10:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"d941056b-c2e7-42e5-965a-a982c20b1169","Qwen3.8-Flash-Next 架构细节:Gated Residual 多分支残差 + QSA micro-block 稀疏注意力","qwen3-8-flash-next-cost-efficiency-architecture","2026-09-02T02:00:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"fdbe1ee2-131c-4632-baf6-03109d7c1814","Qwen3.8-Next 架构论文:125B 参数 6B 激活,1\u002F9 训练 FLOPs 对标 397B 前辈","qwen3-8-flash-next-architecture","2026-09-01T23:15:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"c5413f17-7fd9-4123-92ea-d79293d36b2a","千问 Skill Self-Play 让技能库参与自博弈：Ministral 工具调用从 20.7 跳到 63.6","qwen-skill-self-play","2026-07-28T04:00:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"a9832c84-4903-4d3a-93a4-e0768bcf69b2","SkewAdam 把 MoE LLM 的 optimizer state 砍到 2.6%：让 6.78B 模型塞进 40 GB 显卡","skewadam-moe-optimizer-state","2026-07-23T00:10:00+00:00"]