[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-soft-mtp-kl-gradient-recovery":3,"topics-all":38,"news-related-49972f78-da29-4041-8f1f-c57d9d8ceeac":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"49972f78-da29-4041-8f1f-c57d9d8ceeac","MIT 一行代码修 MTP:八成梯度噪声被滤掉","MTP 已是 DeepSeek、GLM-5、Kimi K3 等的预训练标配,MIT 单作者论文发现其梯度约八成是低信噪比残差,规模越大越吵。Soft-MTP 一行代码回传 KL 自蒸馏信号,167M 到 1.2B 每档都优于标准 MTP;1.2B 微调后 HumanEval pass@64 领先 3.1 个百分点。","MTP(多 token 预测)已经是主流大模型预训练的标配:论文点名 DeepSeek-V3\u002F4、NVIDIA Nemotron 3、GLM-5、Kimi K3 都在用。它的好处一石二鸟——额外的预测头既能改善下游表现,推理时还能当投机解码的起草头。但 10 月 28 日挂上 arXiv 的[单作者论文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.38331)(MIT)指出:标准 MTP 训练目标里埋着一个低信噪比的梯度残差,模型越大,它越拖累学习。\n\n## 占八成的低质梯度\n\n论文把 MTP 的交叉熵梯度做了精确分解:一部分是 KL 自蒸馏项,把 MTP 头往模型自己的下一 token 预测分布上拉——这恰是投机解码想要的对齐信号;另一部分残差,本质是对数据的下一 token 交叉熵梯度,信噪比很低,却把有用的 KL 信号淹掉了。实测上,整个训练过程里残差的信噪比都比 KL 分量低约 4 倍,差距从 167M 的 4.1 倍拉大到 1.2B 的 4.6 倍;而流入共享主干网络的 MTP 梯度里,约 80% 由这个残差贡献。论文还做了单步探针:沿 KL 分量走一步,下一 token 损失下降;沿残差走一步,损失反而上升——它不只是噪声,还在跟主任务对着干。\n\n## 修法:一行代码换干净梯度\n\n直接的思路是最小化显式 KL,但那要物化词表大小的 logit 矩阵,内存吃不消,也跟主流预训练依赖的融合交叉熵内核不兼容。作者提出的 KL 梯度恢复技巧绕开了这一点:当两个分布共享线性输出头时,KL 的梯度可以由两条交叉熵梯度经直通式梯度路由精确组合出来。于是 Soft-MTP 只需一行代码改动:不加参数、不多算 logits、兼容融合内核。在 H200 上实测,吞吐与标准 MTP 只差 0.07%,FLOPs 与显存完全一致;官方仓库带了 float64 单元测试,验证恢复出的梯度与显式算 KL 机器精度等价。DeepSeek 式与 Meta 式 MTP 结构都支持。\n\n## 账本:四档规模都占优,微调最亮眼\n\n论文实测 Soft-MTP 在 167M、332M、689M、1.2B 四档规模(各训 31.5B FineWeb-Edu token)的每一档都优于标准 MTP:零样本平均准确率的增益随规模拉大,小模型约 0.1 分,689M 与 1.2B 约 0.5 分,并在 1.2B 反超纯 NTP 基线(平均 57.31,高于标准 MTP 的 56.80 与 NTP 的 56.75);投机解码验收长度五域平均提升 1.8–2.2%,加速提升 1.4–1.5%。最亮眼的是微调迁移:1.2B 分别在 21B token 的 Stack-Edu Python 与 4.2B token 的 OpenMathInstruct-2 上微调后,Soft-MTP 八格里赢下七格,HumanEval pass@64 领先 3.1 个百分点(43.90 对 40.85),且是唯一从不错破 NTP 底线的目标。对照显式方案 KL-MTP,Soft-MTP 多数指标持平或更好,而后者要多花 1.5% FLOPs、吞吐低 5.6%、显存多 21GB。\n\n## 边界:单作者、自费算力、前沿规模未验证\n\n冷水也要泼:这是单作者预印本,实验主要由作者本人在 Vast.ai 租 8 张 RTX PRO 6000 完成,验证上限是 1.2B 参数、31.5B token,离万亿级前沿训练还远;「免费午餐」是论文自己的说法,数学微调上 KL-MTP 仍占先(GSM8K 50.34 对 49.66)。论文第 7 节还补了一条背景:DeepSeek-V4.1-Flash 已经放弃 MTP、改用训练后得到的 DSpark 投机解码器,MTP 路线本身的前途尚在未定之天。但 KL 恢复技巧的适用面不止 MTP——只要教师与学生共享线性输出头,OPSD 这类训练后自蒸馏场景都能借它省掉昂贵的显式 KL。\n\n对还在用 MTP 的团队,这是一次近乎零成本的工程升级;对所有人,它提醒的是:连「标配」都可能藏着没人算清的梯度级缺陷。四档规模的代码与 checkpoint 均已开源([GitHub](https:\u002F\u002Fgithub.com\u002FJaydenTeoh\u002FSoft-MTP))。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.38331","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"d2aab76c-85b6-478e-a872-7c1058d44b88","en","MIT Fixes MTP: One Line Filters Out Its Noisy 80% Gradient","An MIT paper shows MTP training carries a noisy ~80% residual gradient; a one-line fix removes it free, winning every 167M-1.2B scale.","Multi-token prediction (MTP) has become standard equipment in frontier pre-training — the paper names DeepSeek-V3\u002F4, NVIDIA's Nemotron 3, GLM-5 and Kimi K3 as adopters. The extra heads improve downstream performance and double as speculative-decoding drafters at inference time. But a single-author [paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.38331) from MIT, posted on arXiv on October 28, argues that standard MTP training buries a low signal-to-noise residual in its gradient — one that interferes more as models grow.\n\n## The residual: 80% of the gradient, lowest quality\n\nThe paper decomposes the MTP cross-entropy gradient exactly: a KL self-distillation term that pulls each MTP head toward the model's own next-token distribution (precisely what speculative decoding wants), plus a residual — effectively the next-token cross-entropy gradient against the data — with low SNR that buries the useful signal. Measured across training, the residual's SNR runs about 4x below the KL component, and the gap widens from 4.1x at 167M to 4.6x at 1.2B. Yet that residual carries roughly 80% of the MTP gradient reaching the shared trunk. A unit-step probe makes the split visible: stepping along the KL component lowers the next-token loss; stepping along the residual raises it — it is not just noise, it works against the main task.\n\n## The fix: one line of code\n\nThe obvious alternative — minimizing an explicit KL — would mean materializing vocabulary-sized logit tensors, which is memory-hungry and incompatible with the fused cross-entropy kernels modern pre-training relies on. The author's KL gradient-recovery trick sidesteps it: when two distributions share a linear output head, the KL gradient can be recovered exactly by combining the two cross-entropy gradients through straight-through gradient routing. Soft-MTP is a one-line change: no new parameters, no extra logits, fused-kernel compatible. On an H200, throughput lands within 0.07% of standard MTP, with identical FLOPs and peak memory; the repo ships a float64 test verifying machine-precision equivalence to explicitly computing the KL. Both DeepSeek-style and Meta-style MTP are supported.\n\n## The ledger: wins at every scale, fine-tuning most of all\n\nAcross four sizes — 167M, 332M, 689M and 1.2B (each trained on 31.5B FineWeb-Edu tokens) — Soft-MTP beats standard MTP in the paper's runs at every scale: zero-shot average gains widen with scale — about 0.1 points at small sizes, about 0.5 points at 689M and 1.2B, overtaking the NTP baseline at 1.2B (57.31, vs 56.80 for MTP and 56.75 for NTP); speculative-decoding acceptance length improves 1.8–2.2% averaged over five domains, speedup by 1.4–1.5%. The headline is fine-tuning transfer: after fine-tuning the 1.2B model on 21B tokens of Stack-Edu Python or 4.2B tokens of OpenMathInstruct-2, Soft-MTP takes seven of eight cells, leads HumanEval pass@64 by 3.1 points (43.90 vs 40.85), and is the only objective never below the NTP baseline. Against the explicit KL-MTP variant it matches or exceeds most metrics, while KL-MTP costs 1.5% more FLOPs, 5.6% lower throughput and 21 GB more memory.\n\n## The caveats: single author, rented GPUs, frontier scale untested\n\nThis is a single-author preprint; the experiments were mostly self-funded on 8 RTX PRO 6000 GPUs rented from Vast.ai, the validation ceiling is 1.2B parameters and 31.5B tokens, and \"free lunch\" is the author's own phrasing. KL-MTP still leads math fine-tuning (GSM8K 50.34 vs 49.66). Section 7 adds context: DeepSeek-V4.1-Flash has already dropped MTP for a post-trained DSpark drafter, so MTP's own future is open. The recovery trick, though, applies wherever a teacher and a student share a linear output head — including post-training self-distillation such as OPSD.\n\nFor teams still using MTP, this is a near-zero-cost engineering upgrade; for everyone else, the lesson is that even \"standard equipment\" can hide gradient-level flaws nobody had accounted for. Code and checkpoints for all four scales are open-sourced on [GitHub](https:\u002F\u002Fgithub.com\u002FJaydenTeoh\u002FSoft-MTP).","soft-mtp-kl-gradient-recovery","2026-10-30T17:12:42Z","2026-10-30T17:12:45.520584Z","2026-10-30T17:12:45.520593Z",true,"agent",31,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"a840ee24-8865-4038-986e-1398549feeb0","苹果论文重算循环大模型算力账:停止头说省一半,实测跑 7.6 轮","apple-halting-head-looped-lm-compute-audit","2026-10-26T03:05:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"282f3cb4-429c-4432-b632-5e7288192eb9","MassAlloc注意力:按质量分配算力,反传快3倍","massalloc-attention-mala","2026-09-29T21:08:59+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"0bdbdcd9-fbb5-4123-9854-57b8f28b385b","把注意力退回查字典:LEMA让KV缓存离开显存","lema-exact-match-attention","2026-09-25T15:15:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"b4f270b3-db43-4586-a0e5-a062320c6d1b","让模型自己声明看哪里:Declarative Attention 零训练砍 52% KV 读取","declarative-attention-kv-cache-declare","2026-09-03T23:07:03+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00"]