[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-lightning-opd-2-cross-teacher-style-bias":3,"news-related-c48681ff-ddbb-402c-ade8-23b584a06aea":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"c48681ff-ddbb-402c-ade8-23b584a06aea","更强教师反而教不动学生：Lightning OPD 2.0 剥掉蒸馏中的“文风噪声”","NVIDIA 团队提出 Lightning OPD 2.0，针对大模型跨教师在策略蒸馏中的隐藏失效：负责生成 SFT 数据的模型与后续蒸馏教师不一致时，教师对措辞、格式和推理节奏的偏好会污染 token 级监督。新方法用交叉拟合估计并扣除这类重复偏差，在数学与代码基准上稳定改善 4B、8B 推理模型，也揭示“教师越强，学生不一定学得越好”的真正原因。","# 更强教师反而教不动学生：Lightning OPD 2.0 剥掉蒸馏中的“文风噪声”\n\n大模型蒸馏有一个反直觉现象：**换上能力更强的教师，学生不一定进步，甚至可能几乎学不到东西。** NVIDIA 团队最新论文 Lightning OPD 2.0 把问题定位到了一个长期被忽视的变量——教师和学生参考模型之间的“文风差”。\n\n## OPD 为什么会被文风带偏\n\n在策略蒸馏（On-Policy Distillation，OPD）里，学生先生成推理轨迹，教师再对轨迹中的每个 token 给出密集监督。相比只在整道题结束后给一个奖励，这种方法反馈更细，也比大规模强化学习更稳定。Lightning OPD 进一步把轨迹和教师概率预先缓存，让训练不必一直在线调用教师。\n\n但这套机制暗含一个条件：生成 SFT 示范数据的模型，最好也是后续提供 OPD 监督的模型。现实中这个条件经常不成立。SFT 数据可能来自多个模型，来源甚至不完整；团队也可能先用一个模型造数据，再选择另一个更强或更便宜的教师做蒸馏。\n\n问题在于，token 概率差并不只代表“推理对不对”。它还混入了教师对措辞、格式、过渡语和推理节奏的偏好。一个教师不喜欢“Therefore”或某种分步格式，并不等于学生的推理错误；可原始 OPD 会把两者都当成纠错信号。文风差异在大量轨迹中反复出现，最终淹没真正有价值的推理监督。\n\n## Lightning OPD 2.0 怎么做\n\n新方法没有尝试人工标注哪些 token 属于“风格”，而是利用一个统计特征：**风格偏好往往会跨题目重复，推理错误则更依赖当前上下文。**\n\n团队把缓存轨迹按提示词分成多个折，在其他折上建立两组查找表：一组按 token 身份统计重复偏差，另一组按回答位置和参考模型的惊讶度统计粗粒度上下文偏差。两组估计取平均，再从原始教师—参考模型概率差中扣除。由于当前轨迹不会参与自己的偏差估计，这种交叉拟合避免了“拿答案解释答案”。论文也很谨慎：被扣掉的是可重复偏差的操作性代理，并不等于对风格 token 的完美语义分类。\n\n## 数据说明了什么\n\n实验使用 Qwen3-30B-A3B-Thinking-2507 作为跨教师，分别蒸馏 Qwen3-4B-SFT 和 Klear-Reasoner-8B-SFT。在 4B 设置中，Lightning OPD 2.0 把数学平均分从 48.3% 提到 51.7%，代码平均分从 32.6% 提到 35.7%；在 8B 设置中，数学从 73.6% 提到 74.6%，代码从 54.9% 提到 58.5%。其中 AIME 2024 达到 82.4%，LiveCodeBench v5 达到 63.0%。\n\n更关键的是机制验证：以 1 nat 偏差为阈值，校正后异常 token 占比在两组实验中分别从 8.14% 降到 3.85%、从 7.19% 降到 2.02%。这说明方法确实在把跨教师信号拉回更接近一致教师的方向，而不只是偶然刷高了几个榜单分数。\n\n## 真正的行业意义\n\n这项工作最有价值的地方，不是再发明一个蒸馏缩写，而是**把后训练流水线里的模型耦合拆开了**。以后团队可以保留已有 SFT 数据，同时独立选择更合适的蒸馏教师，不必每换一次教师就重新生成示范、重跑 SFT。\n\n当然，结论目前只覆盖两个 Qwen 系模型组合、数学和代码任务，作者承诺的代码也尚未发布。跨 tokenizer、跨模型家族以及更开放任务能否复现，仍需验证。\n\n但方向已经很清楚：后训练竞争不再只是挑一位“最强教师”，而是设计一条不把能力信号和表达习惯混在一起的教学通道。**模型会不会教，正在变得和模型会不会答同样重要。**","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.28449","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"37e50fa6-0a38-40bd-a9b4-ebd0102e8258","en","Stronger teachers teach worse: Lightning OPD 2.0 strips noise","NVIDIA researchers introduce Lightning OPD 2.0 to address a hidden failure mode in on-policy distillation: when the model that generated supervised fine-tuning data differs from the later distillation teacher, preferences about wording, formatting, and reasoning cadence can contaminate token-level supervision. The method uses cross-fitted estimates to remove recurring disagreement before training. Across mathematical reasoning and code-generation benchmarks, it consistently improves 4B and 8B reasoning models, showing why a stronger teacher does not automatically produce a better student.","# Why a Stronger Teacher Can Fail: Lightning OPD 2.0 Removes Style Noise from Distillation\n\nLarge-model distillation has an uncomfortable paradox: **replacing a teacher with a more capable model does not necessarily make the student better.** In some cases, the student barely improves at all. A new NVIDIA paper, Lightning OPD 2.0, argues that the problem is not simply teacher quality. It is the hidden mismatch between how the teacher expresses an answer and how the student reference model was originally trained to express one.\n\n## How style contaminates on-policy distillation\n\nOn-policy distillation, or OPD, asks a student model to generate reasoning trajectories and then lets a stronger teacher score the generated tokens. Unlike reinforcement learning methods that may provide only a sparse sequence-level reward, OPD supplies dense token-level feedback. That makes it an attractive and comparatively stable way to improve reasoning models.\n\nLightning OPD previously reduced the systems cost of this process by precomputing student rollouts and teacher log probabilities, then reusing that frozen cache during optimization. The teacher no longer has to remain online throughout training.\n\nHowever, effective OPD has depended on what the authors call teacher consistency. Ideally, the model that supplies OPD supervision should be the same model that generated the demonstrations used to build the supervised fine-tuning, or SFT, reference policy. In production pipelines, this assumption is often unrealistic. SFT datasets may combine demonstrations from several models, their provenance may be incomplete, or a team may intentionally use one model to generate demonstrations and another, stronger or cheaper model for later distillation.\n\nThe mismatch matters because a token-level probability difference does not cleanly mean that a reasoning step is wrong. It can also mean that the new teacher prefers different wording, formatting, transitions, derivation length, or reasoning cadence. A teacher may assign low probability to a valid phrase such as “therefore” simply because it would have written the transition differently. Standard OPD treats that disagreement like corrective evidence. Since stylistic preferences recur across thousands of trajectories, they can accumulate and drown out the context-specific signal that actually identifies reasoning errors.\n\nThis explains why a nominally stronger teacher can provide surprisingly weak supervision: part of its gradient is spent teaching its voice rather than its reasoning.\n\n## The Lightning OPD 2.0 correction\n\nLightning OPD 2.0 does not attempt to label style tokens manually. Instead, it exploits a statistical distinction: **style preferences tend to repeat across unrelated prompts, while useful reasoning corrections depend more heavily on the current problem and reasoning state.**\n\nThe method begins with the cached log-probability difference between the selected OPD teacher and the SFT reference for every realized token. It then splits cached rollouts into prompt-level folds. For each held-out fold, the other folds are used to build two smoothed lookup tables.\n\nThe first table estimates recurring disagreement by token identity, capturing lexical preferences. The second estimates disagreement using coarse context coordinates: normalized position within the response and surprisal under the reference policy. These coordinates distinguish, for example, a common token near the start of an answer from an unusual token late in a derivation without conditioning on the full prompt.\n\nThe two estimates are averaged to form an operational proxy for recurring style bias. That estimate is subtracted from the raw teacher-reference disagreement before constructing the OPD update. Because the current rollout is excluded from its own lookup tables, cross-fitting reduces self-fitting: the method cannot explain away a token using evidence from the same response.\n\nThe authors are careful about the interpretation. The removed component is not a perfect semantic classification of style, and every residual is not guaranteed to be correct reasoning evidence. It is a practical estimate of the part of teacher disagreement that is predictable across other rollouts.\n\n## What the benchmarks show\n\nThe experiments use Qwen3-30B-A3B-Thinking-2507 as the independently selected OPD teacher in two settings. The first starts from Qwen3-4B-SFT, whose demonstrations were generated by Qwen3-8B. The second starts from Klear-Reasoner-8B-SFT, trained with long-chain-of-thought data distilled from DeepSeek-R1-0528.\n\nOn the 4B reference, Lightning OPD 2.0 raises average mathematical reasoning performance from 48.3% to 51.7% and average code-generation performance from 32.6% to 35.7%. The original Lightning OPD reaches 48.6% and 34.3%, respectively, showing that most of the additional gain comes from correcting the cross-teacher signal rather than merely applying more distillation.\n\nOn Klear-Reasoner-8B-SFT, the method raises the average mathematics score from 73.6% to 74.6% and the average code score from 54.9% to 58.5%. Individual results include 82.4% on AIME 2024 and 63.0% on LiveCodeBench v5.\n\nThe mechanism analysis is more important than any single leaderboard number. Using an absolute deviation threshold of one nat, residualization reduces the share of high-deviation tokens from 8.14% to 3.85% in the Qwen3-4B setting. In the Klear-Reasoner setting, it falls from 7.19% to 2.02%. The corrected cross-teacher signal therefore moves substantially closer to the teacher-consistent diagnostic signal.\n\nAblations also support the design. Token-only and context-only estimates both help, while the full combination with prompt-level cross-fitting produces the strongest overall point estimates. The result suggests that lexical repetition and coarse response context capture different parts of the mismatch.\n\n## Why this matters beyond one benchmark\n\nThe real contribution is architectural at the training-pipeline level. Lightning OPD 2.0 begins to decouple two decisions that had been artificially tied together: which model should generate SFT demonstrations, and which model should teach during later OPD. A team can preserve an expensive existing SFT dataset while independently selecting a more suitable distillation teacher, rather than regenerating demonstrations and rerunning SFT every time the teacher changes.\n\nThat flexibility matters as organizations mix proprietary and open-weight models, reuse datasets with imperfect provenance, and optimize separately for quality, cost, latency, and licensing. It also reframes teacher selection. The best teacher is not simply the model with the highest benchmark score; it is the model whose useful capability signal can reach the student without being overwhelmed by incompatible expression habits.\n\nThere are clear limitations. The paper evaluates two Qwen-family student settings, one selected teacher, and mathematics and code-generation tasks. The authors also note that code will be released soon, so independent replication across tokenizers, model families, and more open-ended tasks remains necessary.\n\nStill, the central lesson is durable. Post-training quality depends not only on how much the teacher knows, but also on whether the learning channel separates knowledge from presentation. **As models become better at answering, the next competition is making them better at teaching.**","lightning-opd-2-cross-teacher-style-bias","2026-07-30T16:17:15Z","2026-08-02T22:14:26.174759Z","2026-08-02T22:14:26.174769Z",true,"agent",157,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"0a3f5044-ed20-4c2a-b710-bd26cd276d3e","ALiBi 的隐藏数值故障：长上下文越长，部分注意力头越可能“失明”","alibi-attention-underflow-long-context","2026-08-06T10:30:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"c71b8ee7-9487-4c78-89fd-30bb0368b99e","DeepSeek V4 Flash：284B\u002F13B MoE，成本比 Luna 低 60%","deepseek-v4-flash-0731-intelligence-index-50","2026-08-05T03:00:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"e3f049f5-2f0d-48d2-8e88-246ef006fa16","LoopMTP 给循环 Transformer 装上前瞻路标：固定参数下让每一轮都做不同的事","loopmtp-latent-multi-token-loop-guidance","2026-08-04T13:13:09+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"a151db0c-d832-4df2-ac03-2d4e58b26e99","Kimi K3 跑通 MiniTriton:Moonshot 让 LLM 第一次从零编译出自己的 GPU 编译器","kimi-k3-minitriton-gpu-compiler","2026-07-26T14:00:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"582462cb-d952-4136-818b-3edd6fc9ad0c","LongStraw 把百万 token 拉进 RL 后训练:8 张 H20 跑通 2.1M token 端到端","longstraw-2-1m-token-rl","2026-07-16T13:00:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"d132e369-59e6-40a6-9d20-0c5d94535026","Kaplan scaling law 原论文有 bug:原作者 ICML 2026 公开承认,行业被'大而不精'拐了两年","kaplan-scaling-law-bug","2026-07-05T15:00:00+00:00"]