[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-muon-temporary-weight-decay-delayed-effect":3,"topics-all":38,"news-related-3e60e849-882a-4722-9ca8-80324936b001":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"3e60e849-882a-4722-9ca8-80324936b001","撤掉更差,73%后反超:Muon临时权重衰减的延迟账","一篇 Muon 预训练论文发现,临时权重衰减在撤除那一刻比完全不用衰减差 0.26 nats,要到训练走完 73% 才反超,最终反而好 0.0275 nats。论文用范数亏缺与相对步长解释这个延迟效应,并提醒临时干预必须按最终训练预算来评估。","预训练里有一类操作：只在某段窗口里生效，到某个点就撤掉，之后再也不出现。常见做法是临时加一段权重衰减。直觉上窗口一撤就该翻篇，实测却不是这样。\n\n论文《Delayed Effects of Temporary Weight Decay in Muon Pretraining》（arXiv:2611.17876，2026 年 11 月 13 日提交到 arXiv，作者来自 Meta、Mila-Quebec AI Institute、Université de Montréal 与 Samsung SAIL Montreal）给出反直觉的曲线。\n\n## 撤掉那一刻，反而更差\n\n主配置是用 Muon 优化的 decoder-only Transformer，规模从 162M 到 3.4B，数据集是 FineWeb100B（FineWeb 的 100B token 采样）。做法很简单：在一段窗口内打开解耦权重衰减，窗口外关掉。\n\n最有代表性的一组在 1.4B 上：衰减系数 0.15，作用在训练前 44%，schedule 是在 162M 上挑出来直接搬过去的，没重新调参。窗口在第 12,320 步结束时，这条曲线比不用衰减的对照组差 0.26 nats；两者用相同更新规则继续跑，它一路慢慢追，到第 20,440 步（占总训练约 73%）才反超，最终比对照组好 0.0275 nats，也比采样到的最佳常数衰减基线好 0.0264 nats。而在 1.4B 上直接重新采样最优 schedule，只比搬过去那版好 0.0020 nats。\n\n也就是说：撤除时的表现，和最终的结果符号相反。\n\n## 三件事被混在了一起\n\n论文接着拆开这个现象：临时 schedule 有三个可分离维度，却常被当成同一件事。\n\n**开始时间**会翻转符号。在三个规模上，同样是 10% 窗口、同样系数，起始于初始化的版本同时打败「不用衰减」和「最佳常数衰减」两个参照；挪到训练 60% 处才开始的同样窗口，两个参照都打不过。这些窗口都落在恒定学习率阶段、施加同样的乘性收缩，所以是时机效应而非收缩量效应。\n\n**持续时长**比系数更值得调。162M 上跑 16,000 步（算力最优预算的 5 倍），在时长与系数的 4×3 网格里，固定系数改时长，最终改善量在 0.026–0.035 nats 之间动；固定时长改系数，只在 0.007–0.010 nats 之间动。即便把总对数收缩量 D 对齐到 2.1 左右，窗口从 700 步拉长到 3,564 步，改善幅度还是翻了一倍以上——决定结果的是时间分布，不是缩小的总量。\n\n**评估时点**会改写排名。405M 上，按「同一个最终 checkpoint」比较和按「撤除后多训同样步数」比较，早开始窗口的排序是相反的：前者偏好 20% 起始，后者偏好起始于初始化。\n\n## 范数亏缺与相对步长\n\n机制解释落在权重范数上。早期衰减把权重压小，这影响撤除后仍在：1.4B 那组里，撤除时权重范数只剩不用衰减时的 12–16%，到最终 checkpoint 也只有 60–66%。范数小，相对步长（垂直于权重方向的更新量，除以权重范数）就大，而这个差距收敛得很慢。论文给的近似范数动力学递推是：从第 400 步到第 2,800 步，递推式里的范数项涨了约四倍，两次运行之间的差距变化远小于此，相对步长比值只从大约 1.6–1.8 回落到约 1.1。\n\n固定范数重放做了直接检验：把权重范数钉在初始值，只重放实测的相对步长轨迹，早期衰减那条轨迹的最终损失比不用衰减的轨迹低 0.022–0.023 nats，而原始两次 Muon 运行之间的差距是 0.0171 nats。轨迹交换实验更干脆：撤除之后的步子贡献了 0.0221 nats 重放差距中的 0.0181；只在干预期间用早期轨迹、之后换回普通 Muon，只换来 0.0003 nats。反向扰动也对得上：早期抬学习率会抬高范数、压低后续相对步长，三个提学习率版本最终损失都更差。\n\n## 不是权重衰减的专利\n\n为排除「这只是乘性权重衰减的特例」，论文换了个形式完全不同的干预：离对角 Gram 惩罚。162M 上，10% 窗口分别从 0%、20%、60% 开始，最终损失变化是 −0.0147、−0.0080、+0.0209 nats，同样的时间依赖；五次配对运行里，早期 Gram 惩罚比不加正则好 0.0142 nats。分解后，径向分量几乎复现了整个惩罚的效果，切向分量几乎没有影响——起作用的是范数而非方向。\n\n## 跨到 3.4B，以及它没覆盖的部分\n\n3.4B 上结果同向。从 1.4B 迁移的 12,320 步版本最终损失 2.4638，对照组 2.4832，五个任务的平均零样本准确率 66.90 对 65.30；两个常数衰减参照既没降低损失（2.4843 和 2.4895），平均准确率只有 65.68 和 65.69。代价是等待：这组在撤除时比对照差 0.2973 nats，要到第 25,400 步才稳定低于对照。\n\n论文自己划清了边界。多数未做复现的配置只有一个种子；系数不是通用的，机制测量集中在 162M；在 405M 和 1.4B 上，下游评测并未显示临时衰减相对常数衰减的优势；换成 AdamW，162M 上有效而 405M 上无效，现象并非与优化器无关。加 10% 线性 warmup 也消不掉延迟交叉——16,000 步的 162M 实验里，第 8,000 步时还差 0.0237 nats，到第 14,400 步已经好了 0.0373 nats。\n\n## 所以呢\n\n方法论后果很直接：评估一个临时优化干预，得在你真正打算用的训练预算上评估，而不是在撤除那一刻。按撤除点的损失去挑 schedule，可能挑出一个最终更差的方案。\n\n机制上的交叉印证也有人在做。另一支团队在 arXiv:2608.24814 里独立把学习率与参数范数之比当作损失动力学的主导坐标，把「延迟加速」描述为范数控制反复出现的效应——与本文的相对步长解释是同一套语言。Muon 的设计文档（https:\u002F\u002Fkellerjordan.github.io\u002Fposts\u002Fmuon\u002F）解释了为什么范数小会让相对更新变大：更新矩阵经 Newton-Schulz 迭代近似正交化，幅度受控，权重越小，同样大小的更新相对权重越大。\n\n三个方向凑到一起，指向一个不太舒服的结论：训练中途看到的曲线，可能不是最终结果的预告，只是它自己那一段的故事。\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.17876","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"4a70cf86-9f20-46e4-8846-15f74b318719","en","Worse when removed, better when done: delayed Muon decay","Temporary weight decay in Muon pretraining: 0.26 nats worse at removal, overtaking after 73% of training, 0.0275 nats better at the end.","In pretraining there is a class of interventions that only act during a window and then get switched off for good. A temporary weight-decay schedule is the standard example. The intuition is that once the window closes, the story is over. Measured, it is not.\n\nThe paper Delayed Effects of Temporary Weight Decay in Muon Pretraining (arXiv:2611.17876, submitted to arXiv on 13 November 2026, with authors from Meta, Mila-Quebec AI Institute, Université de Montréal and Samsung SAIL Montreal) documents a set of counterintuitive curves.\n\n## At removal, it is worse\n\nThe main configuration trains decoder-only Transformers with Muon at scales from 162M to 3.4B, on FineWeb100B (a 100B-token sample of FineWeb). The method is simple: turn decoupled weight decay on inside a training window and off outside it.\n\nThe most striking case is the 1.4B model: coefficient 0.15, applied over the first 44% of training, with the schedule selected at 162M and transferred over without retuning. When the window ended at update 12,320, this run was 0.26 nats worse than the no-decay reference. Both runs then continued with exactly the same update rule, and the treated run slowly clawed its way back, overtaking the reference at update 20,440 (roughly 73% of training) and finishing 0.0275 nats better, and also 0.0264 nats ahead of the best sampled constant-decay baseline. Sampling the best schedule directly at 1.4B improved on the transferred one by only 0.0020 nats.\n\nIn other words: the measurement at removal and the outcome at the end have opposite signs.\n\n## Three things that get conflated\n\nThe paper then separates this effect: a temporary schedule has at least three separable dimensions, yet they are routinely treated as one thing.\n\n**Start time** flips the sign. At all three scales, a 10% window with the same coefficient that starts at initialization beats both the no-decay and the best-constant references; the same-duration, same-coefficient window that starts at 60% of training loses to both. Because these windows all sit inside the constant-learning-rate phase and apply the same multiplicative shrinkage, this is a timing effect, not a difference in total shrinkage.\n\n**Duration** matters more than the coefficient. At 162M trained for 16,000 updates (five times the compute-optimal budget), in a 4x3 grid of durations and coefficients, varying duration at fixed coefficient moves the final improvement by 0.026-0.035 nats, while varying the coefficient at fixed duration moves it by only 0.007-0.010 nats. Even with total log-shrinkage D matched at about 2.1, stretching the window from 700 updates to 3,564 more than doubles the magnitude of the final improvement. What decides the outcome is the temporal distribution, not how much was shrunk in total.\n\n**Evaluation point** rewrites the ranking. At 405M, comparing at a common final checkpoint and comparing after an equal number of post-removal updates give opposite orderings of the early-start windows: the former prefers the 20% start, the latter prefers the start at initialization.\n\n## The norm deficit and relative steps\n\nThe mechanism lands on weight norms. Early decay shrinks the weights, and the effect survives removal: in the 1.4B run, norms at removal were only 12-16% of their no-decay values, and were still just 60-66% as large at the final checkpoint. Smaller norms mean larger relative steps (the update perpendicular to the weight direction, divided by the weight norm), and that gap converges slowly. The paper's approximate norm-dynamics recurrence runs from update 400 to 2,800, where the recurrence's norm term grows roughly fourfold while the gap between the two runs changes much less, so the relative-step ratio falls only from about 1.6-1.8 to about 1.1.\n\nA fixed-norm replay tests this directly. Holding weight norms at their initial values and replaying only the measured relative-step trajectories, the early-decay trajectory ends 0.022-0.023 nats below the no-decay trajectory, against a 0.0171-nat difference between the two original Muon runs. The trajectory-swap experiment is blunter still: the steps after removal account for 0.0181 of the 0.0221-nat replay difference, whereas replaying only the early-decay trajectory during the intervention and then returning to ordinary Muon buys just 0.0003 nats. The opposite perturbation behaves as predicted too: an early learning-rate boost raises norms and depresses later relative steps, and all three tested boosts finished with higher loss.\n\n## Not a weight-decay monopoly\n\nTo rule out the idea that this is peculiar to multiplicative weight decay, the paper switches to a qualitatively different intervention, the off-diagonal Gram penalty. At 162M, 10% windows starting at 0%, 20% and 60% change final loss by -0.0147, -0.0080 and +0.0209 nats respectively, the same timing dependence; across five paired 162M runs the early Gram penalty improves final loss by 0.0142 nats relative to no regularization. Decomposed, the radial component nearly reproduces the whole penalty while the tangential component has little effect. What does the work is the norm, not the direction.\n\n## Across to 3.4B, and what it does not cover\n\nThe 3.4B results point the same way. The 12,320-update version transferred from 1.4B ends at final loss 2.4638 against 2.4832 for no decay, with mean zero-shot accuracy of 66.90 against 65.30 across five tasks; neither constant reference reduces loss (2.4843 and 2.4895), and their mean accuracies are only 65.68 and 65.69. The cost is waiting: at removal this run is 0.2973 nats behind the reference and only settles below it from update 25,400.\n\nThe paper draws its own boundaries. Most configurations outside the replicated comparisons use a single seed; the coefficients are not universal; mechanism measurements are concentrated at 162M; at 405M and 1.4B the downstream evaluations show no clear advantage of temporary decay over constant decay; and with AdamW the effect holds at 162M but not at 405M, so it is not optimizer-independent. A 10% linear warmup does not remove the delayed crossover either: in a 16,000-update 162M run, temporary decay is 0.0237 nats worse at update 8,000 but 0.0373 nats better by update 14,400.\n\n## So what\n\nThe methodological consequence is direct: evaluate a temporary optimization intervention at the training budget where you actually intend to use the model, not at the moment it is removed. Choosing schedules by the loss at removal can pick one that is worse in the end.\n\nCross-checking of the mechanism is already under way. A separate team reports in arXiv:2608.24814 that loss dynamics are governed primarily by the ratio of learning rate to parameter norm, and describes delayed acceleration as a recurring effect of norm control, which is the same language as the relative-step account here. Muon's own design writeup (https:\u002F\u002Fkellerjordan.github.io\u002Fposts\u002Fmuon\u002F) explains why smaller norms enlarge relative updates: the update matrix is approximately orthogonalized by a Newton-Schulz iteration, so update magnitude is largely controlled, and the smaller the weights, the larger the same-sized update becomes relative to them.\n\nThree directions converge on an uncomfortable conclusion: the curve you see mid-training may not be a preview of the final result at all, just a story about that stretch of training.\n","muon-temporary-weight-decay-delayed-effect","2026-11-16T03:30:00Z","2026-11-16T03:40:49.052279Z","2026-11-16T03:40:49.052288Z",true,"agent",153,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"872f163b-5ed4-4405-91aa-48c0fe34c25f","论文实测:模型在哪一层想通这个词,能预测人读它多久","resolution-depth-word-reading-time-llm","2026-11-22T05:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"0a3f5044-ed20-4c2a-b710-bd26cd276d3e","ALiBi 的隐藏数值故障：长上下文越长，部分注意力头越可能“失明”","alibi-attention-underflow-long-context","2026-08-06T10:30:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"4f5d6e7d-b06c-4331-8e13-b916fa82bd2e","论文实测:70B模型说谎,探针认的是措辞","persona-direction-lie-probe-measures-wording","2026-11-22T09:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"42f5ea83-365c-4a4e-a318-17d7c4c11c15","论文实测:自洽性证书在AIME上24%的题上会误触发","self-consistency-certificates-misfire-correlated-reasoning","2026-11-21T09:30:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"53dd30bb-ba19-45b6-a2de-5011709608b6","论文实测:一段文本认出39%审稿人,靠的是曝光量","frontier-models-name-authors-by-exposure","2026-11-21T01:40:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"c4623b85-37f5-435e-ae98-e67eb72eb1dc","别再逐篇问大模型:AI写程序当评委,每主题0.17美元起","execjudges-llm-written-program-judges-retrieval-eval","2026-11-20T01:40:00+00:00"]