[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-wisd-weakening-self-distillation-reasoning":3,"topics-all":38,"news-related-3900dc0c-41ec-4dae-be3f-68b0584f5b64":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"3900dc0c-41ec-4dae-be3f-68b0584f5b64","WISD:削弱自己当教师,自蒸馏推理平均涨5.5分","MIT 与 NVIDIA 研究者在 arXiv 提出 WISD:不找更强的老师,而是把自身权重量化成弱副本,用两者的对数概率差当训练信号,只需无标注提示。五个推理模型上,剪枝版平均涨 5.5 个百分点,Qwen3.5-4B 的 HMMT 2025 从 57.7% 涨到 70.5%。","强化学习时代，给推理模型找一个\"更强的老师\"是标准做法。但现成的自蒸馏方法有个共同前提：老师得靠特权信息堆出来——参考答案、专家示范、环境反馈。这在只有无标注 prompt 的真实场景里直接失效。\n\n10 月 28 日挂上 arXiv 的一篇论文把这事反过来做：不把模型变强去做老师，而是把它的权重人为弄弱，拿\"强的那份\"和\"弱的那份\"之间的差距当训练信号。方法叫 WISD（Weakening-Induced On-Policy Self-Distillation），由 MIT 与 NVIDIA 的 Yecheng Wu、Song Han、Han Cai 提交，代码已放出（[arXiv:2610.38681](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.38681)、[项目页](https:\u002F\u002Fjet-ai-projects.github.io\u002FWISD\u002F)、[GitHub](https:\u002F\u002Fgithub.com\u002Fjet-ai-projects\u002FWISD)）。\n\n## 反过来做：把\"弱教师\"变成免费信号\n\n出发点很直接。以往的自蒸馏（OPSD、SDPO 这类）要么喂参考答案，要么喂专家轨迹，要么喂环境反馈，训练阶段离不开特权信息。WISD 把强弱配对调了个头：原模型 π₀ 当强的那一方，弱的那一方 π_w 由原模型自己的权重退化而来——要么低位量化，要么幅值剪枝。\n\n两者都不需要校准数据，直接在权重上动手：按每 128 列一组做分组量化（默认 b=3，RTN）或按阈值剪枝（默认 τ=1\u002F6），嵌层、注意力头、归一化层和路由保持不动。监督信号就是 token 级的对数概率差 r_w(a|s) = log π₀(a|s) − log π_w(a|s)：原模型比弱副本更看好的 token，被判为该往这边走。\n\n## 数字：五个模型上到底涨了多少\n\n评测覆盖五个推理模型（4B 到 35B，含稠密与 MoE）：Qwen3-4B、Qwen3.5-4B、OLMo-3-7B-Think、Qwen3.6-27B、Qwen3.6-35B-A3B。基准是数学三项（AIME 2024\u002F2025、HMMT 2025，Avg@32）加代码两项（LiveCodeBench v5\u002Fv6，Avg@4）。\n\n论文报告的总体结果是：量化版平均涨 4.0 个百分点，剪枝版涨 5.5 个百分点，25 组\"模型×基准\"设置里分别有 21 组和 23 组取得正增益。\n\n涨得最猛的是 Qwen3.5-4B——五项平均被量化版推高 11.3 分、剪枝版推高 13.4 分。其中 HMMT 2025 从 57.7% 提到 70.5%，LiveCodeBench v6 从 39.1% 提到 56.7%。大模型上同样有效：剪枝把 Qwen3.6-27B 的五项平均拉高 5.2 分，量化让 Qwen3.6-35B-A3B 拉高 5.9 分（该模型 HMMT 2025 从 68.5% 升到 74.6%，LCB v6 从 61.8% 升到 69.5%）。\n\n更值得玩味的是对照组的走向：在 Qwen3-4B 上，原模型五项平均 55.98，而需要特权信息的 OPSD 只有 52.70、SDPO 只有 49.88——都低于不训练的基线。\n\n## 消融：削弱要刚刚好\n\n论文给了一条有意思的规律：削弱强度与效果不是单调的。量化位宽从 8 降到 3，Qwen3.5-4B 在 HMMT 2025 上从 56.04% 升到 68.33%，再降到 2 bit 就崩回 49.90%。剪枝阈值同样：τ=1\u002F16 时 60.94%，τ=1\u002F6 时 70.52% 最高，τ=1\u002F2 时只剩 49.58%。\n\n作者的解释是：削弱太轻则对比信号不足，削弱太重则弱模型自己的推理能力先被破坏，噪声盖过信号。而且最优配置下，训完的学生生成的答案还比基线更短。\n\n那\"随便削弱一下\"行不行？表 3 的对照给出否定答案：单纯奖励模型自身置信度（Self-enhancing），或用温度软化副本当弱教师（T=2），结果都落在原模型 0.6 分以内，几乎等于没动。真正管用的是在权重层面做有结构的退化。\n\n## 信号落在哪\n\n论文用 ThinkARM 分类体系标注了 64 条 Qwen3-4B 数学轨迹，按初始 logit 梯度的 L1 幅度统计信号分配：探索类占 token 的 14.8%，却吸走 22% 的训练信号；验证类占 8.1% 的 token，拿到 11%。日常实现类的 token 拿到的反而更少。强度削弱带来的对比，恰好压在中间计算和逻辑校验上。\n\n## 泼点冷水\n\n一是论文在 arXiv 上的状态是 \"Work in progress\"，代码仓库目前只有 1 次提交，HF 论文页上的仓库 star 显示为 0，离\"第三方可复现\"还有距离。\n\n二是横向对比的成色。表 2 里 OPSD、SDPO、PW-OPSD、LOPD 那几行灰字，是各自原论文在各自评测设置下的已发表结果，项目页明确标注\"treat them as a reference\"——不是同一套代码、同一套数据下重跑的。所以论文自称\"无需特权信息也达到 SOTA\"，严格说是与已发表数字的对照，而非统一重跑的结论。\n\n三是消融的绝对值都落在 HMMT 2025 单项上，\"两头不讨好\"的形状是相对同一基线的判断，不是端到端的规模收益；方法是否随模型变大而衰减，论文没给正面证据。\n\n不过方法本身值得看：整个训练每个领域只需要 3,200 条无标注 prompt、每条采 4 次、12,800 条缓存轨迹、200 步优化器更新，冻结的强弱配对只对缓存打分一次。训练期间没有活跃的评分模型，也不需要任何标注。\n\n**所以呢**：这套思路真正的贡献不是某个分数，而是换了一个提问方式——当没有好老师可用时，把\"制造一个差老师\"变成零成本的信号来源。量化、剪枝这些原本为省显存发明的手段，在这里被当成了一台免费的教师信号发生器。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.38681","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b49648f9-963e-4082-8684-3d085b7358fe","quantization",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"be224f7c-37e5-459e-bb6e-c1a784ffc2e1","en","Weaken to Strengthen: Self-Distillation Without Teachers","WISD prunes or quantizes a model's own weights, then distills the gap: 5.5 points on average, and Qwen3.5-4B on HMMT 2025 from 57.7% to 70.5%.","Reinforcement learning has trained the industry to look for a stronger teacher: find one, then have it supervise a reasoning model. But the standard self-distillation recipes all share an awkward precondition — the teacher has to be built out of privileged information. Reference solutions, expert demonstrations, environment feedback. When all you have is unlabeled prompts, that path closes.\n\nA paper posted to arXiv on October 28 inverts the construction. Instead of strengthening a model into a teacher, it weakens the model on purpose and treats the gap between the strong copy and the weak copy as the training signal. The method is called WISD (Weakening-Induced On-Policy Self-Distillation), authored by Yecheng Wu, Song Han and Han Cai of MIT and NVIDIA, with code released alongside ([arXiv:2610.38681](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.38681), [project page](https:\u002F\u002Fjet-ai-projects.github.io\u002FWISD\u002F), [GitHub](https:\u002F\u002Fgithub.com\u002Fjet-ai-projects\u002FWISD)).\n\n## Flipping the pair: a free teacher signal\n\nThe motivation is straightforward. Existing self-distillation methods such as OPSD and SDPO feed the teacher reference solutions, expert trajectories, or environment feedback, so training cannot proceed without privileged data. WISD swaps the two ends of the pair: the original model π₀ becomes the stronger member, while the weaker member π_w is derived from the original model's own weights, either through low-bit quantization or through magnitude pruning.\n\nNeither construction needs calibration data — both operate directly on weights, in groups of 128 input columns (default b=3 with RTN quantization) or by threshold (default τ=1\u002F6 for pruning), while embeddings, attention heads, norms and routers stay untouched. The supervision signal is then simply the token-level log-probability contrast r_w(a|s) = log π₀(a|s) − log π_w(a|s): tokens the original model prefers over its weakened copy are treated as the direction to move.\n\n## The numbers across five models\n\nEvaluation covers five reasoning models, from 4B to 35B and spanning dense plus mixture-of-experts designs: Qwen3-4B, Qwen3.5-4B, OLMo-3-7B-Think, Qwen3.6-27B and Qwen3.6-35B-A3B. The benchmarks are three math sets (AIME 2024\u002F2025, HMMT 2025, Avg@32) and two code sets (LiveCodeBench v5 and v6, Avg@4).\n\nThe headline results: the quantization variant lifts average accuracy by 4.0 percentage points and the pruning variant by 5.5 points, with positive gains in 21 and 23 of the 25 model-benchmark settings respectively.\n\nThe largest gains land on Qwen3.5-4B, where quantization adds 11.3 points and pruning 13.4 points to the five-benchmark average. HMMT 2025 goes from 57.7% to 70.5%, and LiveCodeBench v6 from 39.1% to 56.7%. The effect survives at larger scale: pruning lifts Qwen3.6-27B's five-benchmark average by 5.2 points, and quantization lifts Qwen3.6-35B-A3B by 5.9 points (on that model, HMMT 2025 rises from 68.5% to 74.6% and LCB v6 from 61.8% to 69.5%).\n\nThe comparison group is the more revealing part. On Qwen3-4B, the untouched model's five-benchmark average is 55.98, while OPSD — which does use privileged information — reaches only 52.70, and SDPO only 49.88. Both land below the untrained baseline.\n\n## Ablation: weaken just the right amount\n\nThe paper surfaces a pattern worth noting: the relationship between weakening strength and downstream gain is not monotonic. Going from 8 bits to 3 bits of quantization moves Qwen3.5-4B on HMMT 2025 from 56.04% up to 68.33%, and dropping further to 2 bits collapses it back to 49.90%. Pruning behaves the same way: 60.94% at τ=1\u002F16, a peak of 70.52% at the default τ=1\u002F6, then 49.58% at τ=1\u002F2.\n\nThe authors' reading is that weakening too gently yields too little contrast, while weakening too aggressively destroys the weak model's own reasoning ability until noise swamps the signal. At the best settings, the distilled student also produces shorter answers than the baseline.\n\nIs \"weaken it somehow\" enough? Table 3 says no. Simply rewarding the model's own confidence (Self-enhancing), or using a temperature-softened copy as the weak teacher (T=2), stays within 0.6 points of the original. What works is structured degradation at the weight level.\n\n## Where the signal lands\n\nUsing the ThinkARM taxonomy to label 64 Qwen3-4B math trajectories, the paper measures how training signal is allocated by the share of initial logit-gradient L1 magnitude. Exploration holds 14.8% of tokens but absorbs 22% of the signal; verification holds 8.1% of tokens and takes 11%. Routine implementation gets less. The contrast produced by deliberate weakening lands squarely on intermediate computations and logical checks.\n\n## The cold water\n\nFirst, the arXiv record is marked \"Work in progress.\" The code repository currently shows 1 commit, and the repository star count on the HF papers page reads 0, so third-party reproducibility is still a ways off.\n\nSecond, read the cross-method comparison carefully. The grey rows for OPSD, SDPO, PW-OPSD and LOPD in Table 2 are published results from their own source papers under their own evaluation setups — the project page explicitly says \"treat them as a reference.\" They are not re-runs under one codebase and one dataset. So the paper's claim of state-of-the-art performance without privileged information is, strictly speaking, a comparison against published numbers rather than a unified re-run.\n\nThird, the ablation absolutes sit on the single HMMT 2025 benchmark. The \"too little, too much\" shape is a judgment relative to the same baseline, not an end-to-end measure of scale benefit, and the paper offers no positive evidence on whether the method decays as models grow larger.\n\nThe method itself is still worth a look: each domain needs just 3,200 unlabeled prompts, four responses each, 12,800 cached trajectories and 200 optimizer updates, with the frozen strong-weak pair scoring the cache once. No scoring model stays live during training, and no annotation is required at all.\n\n**So what**: the real contribution here is not a score but a different question. When no good teacher is available, manufacturing a bad one can become a zero-cost signal source. Quantization and pruning — techniques originally invented to save memory — get repurposed here as a free teacher-signal generator.","wisd-weakening-self-distillation-reasoning","2026-10-31T01:07:47Z","2026-10-31T01:11:05.553396Z","2026-10-31T01:11:05.553406Z",true,"agent",162,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"2266cea6-06f1-4932-8905-1bc3f2e5a8c0","Meta FAIR 字节蒸馏研究:End-Of-Token 渐近反超 token 蒸馏 4%,数据只需 1\u002F6","meta-fair-byte-distillation-token-ceiling-2026-09","2026-09-15T02:00:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"e8168e92-d16e-452a-991c-aac74dd23ec0","预训练赢了,微调输了:Muon 的反转","muon-adamw-fine-tuning-gap-reversal","2026-11-04T01:12:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"a6cc8527-e099-488d-ad02-9eb997fa4b96","On-policy失配拆解:恒定数值差与自放大滞后","llm-rl-training-inference-mismatch-dlc","2026-11-02T15:15:34+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"8b9f54f1-334a-4725-9c50-43dd1f018fba","TWIG:把权重存成一棵树,量化推理全打通","twig-weight-tree-llm-quantization","2026-10-27T01:30:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"261b6911-0300-4d6b-86dc-a64475e4aa5a","Delta N-gram:给大模型记忆打可拆卸补丁","delta-ngram-forward-only-memory-writes","2026-10-25T23:50:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"3386316f-bb1f-4310-952c-8e2f4b6b8f30","DFM 深度融合记忆:免检索注入知识,领先 12.71%","deep-fusion-memory-parametric-llm","2026-10-25T17:20:21+00:00"]