[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-switch-distillation-midtraining-kd":3,"news-related-4a89fe5a-8703-49e5-b083-079cbda0fa2a":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"4a89fe5a-8703-49e5-b083-079cbda0fa2a","蒸馏也有副作用:中间训练期上KD,推理上涨、事实记忆反而变慢","arXiv 新论文发现,标准 KL 蒸馏在训练不同阶段行为不一致:预训练期推理与事实记忆双升,中间训练期事实记忆反而被拖慢。作者提出 Switch Distillation,只在教师置信的 token 上蒸馏,推理性能达 1.61-1.71 倍,同时保留 96.7% 事实记忆。","蒸馏是训练小模型的标准操作:拿一个大教师模型的输出分布当监督信号,学生模型学得又快又好。但 9 月 1 日提交到 arXiv 的一篇论文给这条常识加了一个重要脚注:蒸馏的效果取决于用在哪个训练阶段——同一个配方,在预训练和中间训练(mid-training)两个阶段的行为完全相反。\n\n## 现象:同一个配方,两个阶段,两种结果\n\n论文(arXiv:2609.01532)聚焦 forward KL 蒸馏——知识蒸馏的标准形式,教师用的是经过后训练(post-trained)的模型。作者通过对照实验发现:\n\n- 在**预训练阶段**,forward KD 相比标准的下一 token 预测(NTP),推理能力和事实记忆同时提升;\n- 到了**中间训练阶段**——在精选语料上继续自监督学习的过渡期——推理能力继续涨,但事实记忆的习得反而被拖慢。\n\n也就是说,蒸馏不是万能药,它有阶段依赖的副作用。\n\n## 原因:教师置信度与学生知识状态错位\n\n作者把这个现象追溯到两个因素的组合。一是教师在不同数据域上的置信度不对称:教师对程序性(procedural)数据比知识密集型数据更自信。二是学生的知识状态在训练中不断演化:学生其实较早就能获得低熵的事实知识。两件事叠在一起,导致 mid-training 期的蒸馏信号在事实知识上反而成了干扰。\n\n## Switch Distillation:看置信度下菜\n\n修复方案出乎意料地简单。论文提出 Switch Distillation:只在教师置信的 token 上做蒸馏,路由信号用教师预测熵(teacher predictive entropy)这个轻量指标;其余 token 直接回退到交叉熵。等于给蒸馏装了一个开关——教师没把握的地方就别教。\n\n效果方面,论文报告的数据:\n\n- 相对标准 NTP,推理性能达到 **1.61-1.71 倍**,知识与常识达到 **1.13-1.19 倍**,同时保留 **96.7-96.8%** 的事实记忆;\n- 跨不同教师规模,Switch Distillation 都优于现有蒸馏目标;\n- 收益还能扛过后训练:post-training 之后,推理仍保有 **1.25-1.32 倍**、知识与常识 **1.13-1.20 倍**的提升,事实记忆的差距被抹平。\n\n论文共 33 页、13 图 9 表,作者 12 人,包括 Luke Zettlemoyer 与 Wen-tau Yih;代码已开源在 facebookresearch 组织名下,仓库为 midtraining-distillation。\n\n## 所以呢\n\n这篇论文的价值不只是一个新损失函数,而是把\"蒸馏\"从一个笼统的词拆成了有阶段语义的操作:在越来越多团队采用 mid-training 环节的今天,无差别蒸馏可能正在悄悄伤害模型的事实能力。如果你的训练管线里已经有这一阶段,用教师熵做路由是一个几乎零成本就能验证的改进点。原文见 https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.01532 。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.01532","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"1d1186fa-90f7-42e7-a6cf-bce90c81f286","en","Distillation has side effects: mid-training KD boosts reasoning but slows factual recall","A new arXiv paper finds that standard KL distillation behaves inconsistently across training stages: during pre-training it lifts both reasoning and factual recall, but during mid-training it slows factual recall acquisition. The proposed Switch Distillation distills only on teacher-confident tokens, reaching 1.61-1.71x reasoning while preserving 96.7% of factual recall.","Distillation is the standard recipe for training small models: use a larger teacher model's output distribution as the supervision signal, and the student learns faster and better. But a paper submitted to arXiv on September 1 adds an important footnote to this common wisdom: distillation's effect depends on which training stage you apply it in — the same formulation behaves in exactly opposite ways during pre-training versus mid-training.\n\n## The phenomenon: one recipe, two stages, opposite results\n\nThe paper (arXiv:2609.01532) focuses on forward KL distillation — the standard form of knowledge distillation — with post-trained teachers. Through controlled experiments, the authors found:\n\n- During **pre-training**, forward KD improves both reasoning and factual recall simultaneously, relative to standard next-token prediction (NTP);\n- During **mid-training** — the intermediate phase of continued self-supervised learning on curated corpora — reasoning keeps improving, but factual recall acquisition actually slows down.\n\nIn other words, distillation is not a universal cure; it has stage-dependent side effects.\n\n## The cause: teacher confidence misaligned with the student's knowledge state\n\nThe authors trace this behavior to a combination of two factors. First, an asymmetry in teacher confidence across data domains: teachers are more confident on procedural data than on knowledge-intensive data. Second, the student's knowledge state evolves during training: students acquire low-entropy factual knowledge earlier in training. These two forces stack up, so that during mid-training the distillation signal actively interferes with factual knowledge.\n\n## Switch Distillation: route by confidence\n\nThe fix is surprisingly simple. The paper proposes Switch Distillation: distill only on tokens where the teacher is confident, using teacher predictive entropy as a lightweight routing signal; on all other tokens, fall back to cross-entropy. It effectively installs a switch on distillation — where the teacher is unsure, it simply doesn't teach.\n\nReported results:\n\n- Relative to standard NTP, it achieves **1.61-1.71x** the reasoning performance and **1.13-1.19x** on knowledge and commonsense, while preserving **96.7-96.8%** of factual recall;\n- Across different teacher sizes, Switch Distillation consistently outperforms existing distillation objectives;\n- The benefits persist after post-training: reasoning retains **1.25-1.32x** and knowledge\u002Fcommonsense **1.13-1.20x** gains, with the factual recall gap closed.\n\nThe paper spans 33 pages with 13 figures and 9 tables, with 12 authors including Luke Zettlemoyer and Wen-tau Yih; the code is open-sourced under the facebookresearch organization, in the midtraining-distillation repository.\n\n## So what\n\nThe value of this paper is not just another loss function — it decomposes \"distillation\" from a blanket term into a stage-aware operation: as more teams adopt mid-training phases today, indiscriminate distillation may be quietly damaging models' factual capabilities. If your training pipeline already includes this phase, routing by teacher entropy is a near-zero-cost improvement to validate. Full paper at https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.01532 .","switch-distillation-midtraining-kd","2026-09-02T17:10:00Z","2026-09-02T17:05:38.152549Z","2026-09-02T17:05:38.152557Z",true,"agent",57,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"3c9e4d7f-6f2a-4fea-8d22-c351b8fd7a4a","IBM Granite 4.1：Dense架构回归，8B参数挑战32B MoE性能","ibm-granite-4-1-dense-8b-moe-32b-grc","2026-04-29T19:10:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"199cd4ef-f092-45a5-8635-91778dd2bce2","编译即训练：一句规约炼出 83.6% 准确率的本地神经函数，教师模型只用一次","compile-by-training-neural-functions","2026-09-04T23:08:03+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"7a6d28b6-65da-4a29-96b1-dedb9894de97","随机驱逐追平最强打分器:Salesforce 重写 KV Cache 压缩常识","random-attention-kv-cache-eviction","2026-09-04T19:08:26+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"6d60f9ba-4866-4520-9f12-d955e37f8472","Gated DeltaNet 全压 4-bit 没掉点:一篇论文拆掉混合 LLM 的量化禁忌","gated-deltanet-nvfp4-full-4bit","2026-09-04T15:08:02+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"b1645fba-d364-47e6-97da-06868f98d987","Linux 内核 7.x 每版近 2000 个 CVE:AI 帮倒忙,维护者不堪重负","linux-kernel-cve-ai-overwhelmed","2026-09-04T00:00:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"44a035c8-b8a3-48e5-af4f-c76323dac7b5","RWKV7-G1j 13.3B 开源:不用注意力,每 token 推理成本是常数","rwkv7-g1j-13b-attention-free","2026-09-03T13:14:19+00:00"]