[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-muon-adamw-fine-tuning-gap-reversal":3,"topics-all":38,"news-related-e8168e92-d16e-452a-991c-aac74dd23ec0":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"e8168e92-d16e-452a-991c-aac74dd23ec0","预训练赢了,微调输了:Muon 的反转","卡内基梅隆大学团队做了近 600 次预训练与 2.9 万次微调,报告一个反直觉结果:预训练损失更低的 Muon 在数据足够大之后微调反而输给 AdamW。反转大约发生在 2.4B token 处,数学、指令跟随和音乐三类任务上都能看到,并且模型越大,这个反转来得越早。说明预训练损失只是越来越糟的代理指标。","Muon 是这两年预训练最热的优化器。论文的相关工作提到，Kimi、GLM-4.5 这些前沿模型已经用 Muon 或其变体做过预训练。但围绕优化器优劣的讨论几乎都停在一件事上：预训练损失。谁的 loss 更低，谁就是更好的优化器。\n\n卡内基梅隆大学的一篇新论文把这条前提本身当成了研究对象。论文报告的结果有点反直觉：当预训练数据量足够大时，预训练损失更低的那一方，微调之后反而更差。实验规模近 600 次预训练加 2.9 万次微调，反转点在 2.4B token 处，模型越大反转来得越早。\n\n## 先把\"赢\"这件事定义清楚\n\n论文不比单次微调的分数，而是把每个基座能到达的结果收成一张前沿图：横轴是微调后的预训练损失（忘掉了多少），纵轴是微调损失（学会了多少）。学得越多忘得越多，所以只取左下角那条帕累托前沿。两个前沿用一个带符号的量比较，正数代表 AdamW 压过 Muon，负数代表反过来。\n\n数据预算用 token 与参数之比 TPR 衡量，即每个参数摊到多少预训练 token。关键指标是交叉 TPR——AdamW 前沿开始压过 Muon 所需的最小 TPR。\n\n实验用的是 OLMo 风格解码器模型，30M 到 600M 五档参数量，语料是 DCLM，tokenizer 用 OLMo 2。微调侧覆盖 GSM8K、StackMathQA、Alpaca 和 MusicPile 四个数据集，每个基座微调 20M token。\n\n## 反转点在哪\n\n在 StackMathQA 上、数据量最小的那档（2.4B token），Muon 在前沿图上直接压过 AdamW，差距为负。越过这个量级方向反过来：AdamW 拿到更低的微调损失，同时忘得更少，差距转正。\n\n论文把反转拆成两个同时发生的变化。一是 Muon 的预训练损失优势随模型变大而收窄；二是同样固定 TPR 和微调损失，Muon 相对 AdamW 忘得越来越多。两边叠加，AdamW 就能在更小的数据预算上完成反超。\n\n规模效应体现在交叉 TPR 上：StackMathQA 上，30M 模型要 320 TPR 才翻转，600M 模型只需要 10 TPR。模型越大，AdamW 翻盘越早。\n\n## 换微调优化器救不回来\n\n自然的怀疑是：差异也许来自微调配方而不是基座本身。论文做了对照。在 GSM8K 上，AdamW 预训练的基座无论用哪个优化器微调，AdamW 微调的前沿在每个预算下都压过 Muon 微调；而 Muon 预训练的基座，两种微调的前沿挨得很近，没有稳定赢家。更关键的是，1.2B token 时 Muon 预训练的两条前沿还在 AdamW 预训练之下，19.2B token 起 AdamW 预训练加 AdamW 微调的前沿同时压过另外两条。\n\n结论是差异出在起点，不在配方。\n\n用 replay 或 EWC 这类正则化做微调，两个前沿都能少忘一些，交叉点还能往外推——在 Alpaca 和 StackMathQA 上从 80 TPR 推到 320 TPR——但大预算下 AdamW 的前沿依然更好，只有 EWC 加 StackMathQA 在 2,560 TPR 这档例外，差距又变回负的。\n\n## 机制：把遗忘换算成一个可测的量\n\n论文的解释路径是：微调本质上是给输出分布加扰动，所以要看基座对输出扰动的鲁棒性。\n\n对应的指标叫 margin-to-spread ratio（MSR）——正确 logit 与错误 logit 平均值之间的差距，再除以错误 logit 的离散程度（以它们的标准差度量）。MSR 越大，正确答案被撑得越开，对扰动越不敏感。在真实语言模型上，论文在留出的 DCLM 文本里逐 token 算 MSR，取正确预测位置的中位数，覆盖 30M 到 100M 三档：预算小的时候 Muon 更大，预算足够大时 AdamW 更大，方向和微调结果的翻转一致。\n\n理论部分用线性联想记忆做模型，把每个事实表示成键值对，用一个权重矩阵关联起来。论文证明，从零初始化出发做一次更新后，梯度下降与 Muon 的 MSR 之比随数据预算上升，主项是 3π\u002F8 乘以（1 减去 9 除以 10γ）。3π\u002F8 约等于 1.178，在 γ 约 5.95 处越过 1，论文在 γ 大于等于 16 的区间给出严格证明。模拟实验显示多步训练后趋势依旧。\n\n有个必须点明的口径问题：理论比的是梯度下降和 Muon，不是 AdamW；线性联想记忆只是玩具模型，论文自己说明这不构成对 AdamW 的直接证明。\n\n## 边界与相反证据\n\n论文与另一条线索的结论并不一致。Ruan 等人的工作《Muon Learns More Robust and Transferable Features than Adam》报告 Muon 预训练的模型在 Alpaca、Dolly 和 WizardLM 上全量微调后迁移性更好。CMU 这篇的作者认为两者在小数据区间一致，但随着预训练数据增长结果会变；他们也承认两篇研究的模型和下游评测不同，不算对照复现。\n\n论文自己划的边界还有几条：微调分布贴近预训练分布，本质就是继续预训练，结论未必适用；MSR 只是候选的优化目标，论文明确表示没有把它确立为经过验证的训练目标，也没有证明它就是反转的原因；实验规模到 600M 参数为止，交叉 TPR 的具体数值依赖任务，可靠的是趋势方向。\n\n## 所以呢\n\n这篇论文值得带走的不是\"Muon 不行\"或者\"AdamW 要反超\"，而是一个方法论提醒：预训练损失衡量的是模型有多会预测训练分布，不是它有多容易被改造。而且随着数据预算变大，这个代理指标还在变差。\n\n论文结论一节把话说明白了——超参数、数据、架构这些预训练决策现在大多按预训练损失来选，但更低的预训练损失并不保证模型微调后更好。论文给出的方向是把可适应性当作和损失并列的优化目标，MSR 是一个候选指标；至于它能不能真的拿来做训练目标，论文没有给配方。\n\n（全文与图表见 [arXiv 论文页](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.03633)）","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.03633","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"1c6d99f9-d663-4835-9a79-04ef54009ae1","en","AdamW Overtakes Muon After Fine-Tuning as Data Grows","A CMU study across 600 pretraining runs and 29,000 fine-tuning runs finds AdamW overtaking Muon after fine-tuning past 2.4B pretraining tokens.","Muon has been the optimizer everyone’s pretraining runs have been chasing, and the paper’s related-work section notes that frontier models such as Kimi and GLM-4.5 were pretrained with Muon or its variants. But nearly every argument about which optimizer is better has stopped at one measure: pretraining loss. Whoever posts the lower loss is treated as the better optimizer.\n\nA new paper from Carnegie Mellon turns that premise into the research question. The result the authors report is counter-intuitive: once the pretraining data budget grows large enough, the optimizer with the lower pretraining loss ends up worse after fine-tuning. The experiment scale is nearly 600 pretraining runs and 29,000 fine-tuning runs, the reversal point sits at 2.4B tokens, and the reversal arrives earlier as models get bigger.\n\n## Defining what winning means\n\nThe paper does not compare single fine-tuning scores. Instead, for each base model it collects every reachable outcome into a frontier: fine-tuned pretraining loss on one axis (how much was forgotten), fine-tuning loss on the other (how much was learned). Since learning more costs forgetting, only the lower-left Pareto frontier is kept. The two frontiers are compared with a single signed quantity, positive meaning the AdamW frontier dominates and negative meaning Muon’s does.\n\nThe data budget is measured as TPR, the token-parameter ratio, or how many pretraining tokens land on each parameter. The key metric is the crossover TPR: the ratio at which the AdamW frontier first overtakes the Muon one.\n\nThe models are OLMo-style decoder-only transformers at 30M, 60M, 100M, 300M and 600M parameters, trained on DCLM with the OLMo 2 tokenizer. The fine-tuning side covers four datasets, GSM8K, StackMathQA, Alpaca and MusicPile, with every base model fine-tuned for 20M tokens.\n\n## Where the reversal lands\n\nOn StackMathQA at the 2.4B-token budget, Muon Pareto-dominates AdamW and the gap is negative. Past that point the direction flips: AdamW reaches a lower fine-tuning loss while forgetting less, and the gap turns positive.\n\nThe authors split the reversal into two changes that happen at once. Muon’s pretraining-loss advantage narrows as model size grows, and at a fixed TPR and fixed fine-tuning loss, Muon forgets progressively more than AdamW. The two push in the same direction, which is what lets AdamW overtake at a smaller data budget.\n\nThe scale effect shows up in the crossover TPR: on StackMathQA, 30M models need 320 TPR to flip, while 600M models need only 10 TPR. The bigger the model, the earlier AdamW takes the lead.\n\n## Swapping the fine-tuning optimizer does not rescue it\n\nA natural suspicion is that the difference comes from the fine-tuning recipe rather than the starting checkpoint. The paper runs that control. On GSM8K, the AdamW-pretrained base has its AdamW-fine-tuned frontier above the Muon-fine-tuned one at every budget, while the two fine-tuning choices on a Muon-pretrained base sit close together with no consistent winner. More telling: at 1.2B tokens both Muon-pretrained frontiers are still below the AdamW-pretrained one, and from 19.2B tokens onward the AdamW-pretrained, AdamW-fine-tuned frontier dominates both. The difference lives in the starting point.\n\nRegularizing the fine-tuning with replay or EWC reduces forgetting for both sides and can push the crossover outward, from 80 TPR to 320 TPR on Alpaca and StackMathQA, but AdamW’s frontier is still better at large budgets on every dataset under both regularizers, with one exception: under EWC on StackMathQA the gap turns negative again at 2,560 TPR, the top of the range.\n\n## The mechanism, expressed as a measurable quantity\n\nThe explanatory route runs through perturbation. Fine-tuning is treated as perturbing the output distribution, so what matters is how robust the base model is to such perturbations.\n\nThe metric is the margin-to-spread ratio (MSR): the gap between the correct logit and the average incorrect logit, normalized by the spread of the incorrect logits, with the standard deviation of the incorrect logits standing in for that spread. A larger MSR means the correct answer is held further clear of its competitors and the prediction tolerates perturbation better. In real language models the authors compute the MSR token by token on held-out DCLM text and take the median over correctly predicted positions, across the three model sizes spanning 30M to 100M: Muon holds the larger MSR at small budgets, AdamW at large ones, the same direction as the fine-tuning reversal.\n\nThe theory uses a linear associative memory, with each fact a key-value pair and a single weight matrix trained to associate them. The authors show that from zero initialization, after one update, the ratio of gradient descent’s MSR to Muon’s grows with the data budget, with leading term 3π\u002F8 times (1 minus 9 over 10γ). Since 3π\u002F8 is about 1.178, the leading term crosses one at γ near 5.95, and the paper gives a proof for the range γ at or above 16. Simulations over many steps keep the trend.\n\nOne scope point has to be stated plainly: the theory compares gradient descent against Muon, not AdamW, and the linear associative memory is a toy model. The authors do not present it as a direct proof about AdamW.\n\n## Limits and a conflicting result\n\nThe paper does not line up with one line of work. Ruan et al., in “Muon Learns More Robust and Transferable Features than Adam”, report better transfer for Muon-pretrained models after full fine-tuning on Alpaca, Dolly and WizardLM. The CMU authors read their own low-budget result as consistent with that work while showing the comparison changes as pretraining data grows, and they note that the two studies use different models and downstream evaluations, so this is not a matched replication.\n\nThe authors draw their own boundaries. A fine-tuning distribution close to the pretraining distribution amounts to continued pretraining, where the picture may differ. The MSR is a candidate objective rather than a validated training target, and the paper states plainly that it does not establish MSR as a validated training objective or show that MSR causes the reversal. The experiments top out at 600M parameters, the specific crossover TPR depends on the dataset, and the reliable part is the direction of the trend.\n\n## So what\n\nThe takeaway is not that Muon is bad or that AdamW is coming back. It is methodological: pretraining loss measures how well a model predicts its training distribution, not how easily it can be adapted. And as the data budget grows, that proxy gets worse.\n\nThe conclusion section spells it out: hyperparameters, data and architecture are mostly selected by pretraining loss, yet a lower pretraining loss does not by itself yield a model that adapts better. The suggested direction is to treat adaptability as a target alongside loss, with the MSR as one candidate. Whether that indicator can actually be optimized against is left open, and the paper supplies the question rather than a recipe.\n\n(Full text and figures: [arXiv paper page](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.03633))","muon-adamw-fine-tuning-gap-reversal","2026-11-04T01:12:00Z","2026-11-04T01:12:32.169334Z","2026-11-04T01:12:32.169344Z",true,"agent",79,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"bb244989-00b5-4b67-b70a-5316452ee44e","把 RLVR 的\"零梯度\"浪费变成有效信号：Mistral 联合剑桥用 Reasoning Arena 让推理训练提速 41%","reasoning-arena-mistral-cambridge-41pct","2026-06-08T00:00:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"94de69e8-d5b3-4199-ac19-aa7ca4e2299f","视频RL的奖励自己调权重:EmoReward公开代码","emoreward-video-rl-self-recalibrating-rewards","2026-11-06T03:15:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"37e0486a-225d-4b44-b189-206bd756210c","多轮 Agent 强化学习提升了什么:46.8% 的真相","multi-turn-agentic-rl-budget-dependence","2026-11-06T01:18:33+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"99c1eb7b-9226-4d61-afdb-5d54f920a85c","研究品味实测:AI 智能体的想法匹配只有人类的 18%","ideataste-llm-research-taste-benchmark","2026-11-04T11:13:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"a43599f1-d635-4174-9a25-6e1031c497b1","长上下文崩掉的真凶不是长度，是内容竞争","context-rot-attention-competition-not-length","2026-11-04T09:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"65de4df2-898d-4202-8f69-ac03e203e34c","0.8B 模型算对 220 题:正确性由 Lean 证明","sketchpad-certified-arithmetic-small-vlm","2026-11-04T07:00:00+00:00"]