[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-air-elr-schedule-weight-normalized":3,"topics-all":38,"news-related-2f3e98d3-1168-470f-8cf3-6b0d09ee0ada":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"2f3e98d3-1168-470f-8cf3-6b0d09ee0ada","1280亿token实测:cosine调度输给1\u002F√t衰减","权重归一化训练的预训练收益另有其主。字节 Seed 团队论证:收益由诱导的有效学习率轨迹驱动,匹配该轨迹能让归一化与权重衰减的训练动力学基本重合。据此提出的 AIR 调度在 1\u002F√t 衰减上叠加仿射调整,在 1280 亿 token 混合专家模型预训练中,终端损失比 cosine 调度低约 0.01。","cosine 退火几乎是预训练学习率调度的默认答案,大家调参时盯着的都是名义学习率曲线。一篇新论文把提醒直接写进了标题:Your LR Schedule May Not Be Principled——你的学习率调度,可能没道理。问题不在曲线选得不够精细,而在调度的对象本身就选错了。标题里的 Principled 一词,针对的正是“人人都在用、却说不清为什么”的现状:调度曲线长期靠惯例和消融实验挑出来,而不是从一个明确的调度对象出发做设计。\n\n## 权重归一化的收益,另有其主\n\n背景是权重归一化(weight-normalized,WN)训练正在成为传统权重衰减(weight decay,WD)的替代方案,近期研究报出了可观的预训练 loss 收益。这篇论文(arXiv:2610.40362,作者 Qifan Yu、Xinyu Ma、Xingyan Bin、Di He;Hugging Face 论文页将团队标注为 ByteDance Seed)先把刀砍在归因上:这些收益主要由 WN 诱导出的**有效学习率(ELR)轨迹**驱动,而不是归一化本身。证据是,一旦把两条 ELR 轨迹匹配起来,WN 和 WD 的训练 loss 动力学会大体坍缩成同一条曲线。\n\n换句话说,WN 与 WD 之争里真正起作用的变量,是被归一化“顺手”改写的有效学习率;调度设计应该直接看准 ELR,而不是名义 LR。归因一换,两个阵营的对立叙事就都站不稳了:WD 用户不必羡慕 WN 的数字,WN 用户也不该把收益记在归一化头上。\n\n## AIR:理论推导的 1\u002F√t,加上一层仿射\n\n沿这个原则,论文提出 AIR(Affine Inverse-square-Root decay):在理论上推导出的 1\u002F√t 衰减之上,叠一个仿射调整,使得在有限训练步数内,峰值学习率和终端学习率都能落在预设值上。论文还进一步识别出收益的关键来源是**晚期 ELR 调度**——因为共享的晚期调度会稀释早期 LR 选择的影响,收官阶段的曲线形状才是大头。这也顺带解释了一种常见体感:早期 warmup 的具体形状常常没想象中关键,只要晚期把 ELR 收得足够好,前面选择的差异会被摊薄。调参的优先级因此要倒过来——先定收官曲线,再回头谈起点。\n\n## 1280 亿 token 的 MoE 预训练实测\n\n实验规模不小:128 B-token(即 1280 亿 token)的 Mixture-of-Experts 预训练。按论文口径,AIR 在训练 loss、下游表现和超参鲁棒性三个维度上超过 cosine 及其他候选调度;在累计学习率对齐的条件下,终端训练 loss 比 cosine 低约 0.01。更有说服力的是反向验证:把同一条 ELR 轨迹套回 WD 训练,能复现对应的训练动力学——收益不是 WN 专属,ELR 才是那个“有原则”的调度对象。\n\n当然,边界也清楚:这是单一团队的 1280 亿 token 实验,HF 论文页发布一天只有个位数的赞,第三方复现还没有出现,现在就说“cosine 可以退场”为时尚早;但“调调度先看 ELR”这个视角,值得每个调参的人放进工具箱。\n\n## 所以呢\n\n这篇论文真正搬动的不是某条具体调度曲线,而是“调参对象”本身:名义 LR 是你写进配置文件的数字,ELR 才是权重真正感受到的步长。下次复现别人配方失败,先别急着怀疑随机种子——把两边的有效学习率轨迹画出来对一对,可能是更该先做的事。\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.40362","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"a00599f9-255f-47dc-8eb1-fea084808efa","en","AIR Beats Cosine: ELR Is the Right Object for LR Scheduling","Weight-normalized gains trace to the effective learning rate trajectory. AIR schedule beats cosine by ~0.01 loss on 128B-token MoE pretraining.","Cosine annealing is close to a default answer for learning-rate scheduling in pretraining, and when people tune, they stare at the nominal learning-rate curve. A new paper puts the warning right in its title: Your LR Schedule May Not Be Principled. The problem is not that your curve is insufficiently refined — it is that the object being scheduled was chosen wrong in the first place. The word Principled in the title takes aim at a familiar state of affairs: schedule curves have long been picked by convention and ablation, not designed from a clearly defined scheduling object.\n\n## The gains from weight normalization have another source\n\nThe backdrop: weight-normalized (WN) training is emerging as an alternative to conventional weight decay (WD), with recent studies reporting substantial pretraining loss gains. The paper (arXiv:2610.40362, by Qifan Yu, Xinyu Ma, Xingyan Bin, and Di He; the Hugging Face paper page lists the team under ByteDance Seed) first cuts into the attribution: those gains are driven primarily by the **effective learning rate (ELR) trajectory** that WN induces, not by normalization itself. The evidence: once the two ELR trajectories are matched, the training loss dynamics of WN and WD largely collapse onto the same curve.\n\nIn other words, the variable that actually matters in the WN-versus-WD debate is the effective learning rate, quietly rewritten by normalization; schedule design should target ELR directly rather than the nominal LR. Once the attribution shifts, both camps' rival narratives wobble: WD users need not envy WN's numbers, and WN users should not credit normalization for them.\n\n## AIR: a theoretically derived 1\u002F√t plus an affine adjustment\n\nFollowing that principle, the paper proposes AIR (Affine Inverse-square-Root decay): a theoretically derived 1\u002F√t decay with an affine adjustment on top, so that over a finite training horizon both the peak and the terminal learning rate land on prescribed values. The authors further identify **late-stage ELR scheduling** as the key source of the gains — a shared late schedule attenuates the effect of earlier LR choices, so the shape of the endgame curve carries the most weight. This also explains a common piece of folk experience: the exact shape of early warmup often matters less than assumed, because a sufficiently good late-stage ELR dilutes the differences of earlier choices. Tuning priorities invert accordingly — fix the endgame curve first, then work back to the start.\n\n## Tested on 128 B-token MoE pretraining\n\nThe experiment is not small: 128 B-token Mixture-of-Experts pretraining. Per the paper, AIR outperforms cosine and other candidate schedules on training loss, downstream performance, and hyperparameter robustness; at matched cumulative learning rate, terminal training loss is about 0.01 lower than cosine. More persuasive is the reverse check: applying the same ELR trajectory back to WD training reproduces the corresponding dynamics — the benefit is not exclusive to fixed-norm training, and ELR is the principled object to schedule.\n\nThe boundaries are equally clear, though: this is a single team's 128 B-token experiment, the HF paper page has gathered only single-digit upvotes a day after release, and no third-party replication has appeared — too early to declare cosine retired. But \"look at ELR before touching the schedule\" is a lens worth keeping in every tuner's toolbox.\n\n## So what?\n\nWhat this paper really moves is not one specific schedule curve but the object of tuning itself: the nominal LR is the number in your config file, while ELR is the step size your weights actually feel. Next time a reproduction of someone's recipe fails, before blaming the random seed — plot both sides' effective learning-rate trajectories and compare. That is probably the earlier thing to check.\n","air-elr-schedule-weight-normalized","2026-10-31T21:12:47Z","2026-10-31T21:12:56.466242Z","2026-10-31T21:12:56.466252Z",true,"agent",289,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"530e5aa5-2026-4c30-b4e6-421caca907b2","Transformer 提前罢工:13 个基座模型跟不住引用链,一个 rank-8 LoRA 修好","tiny-lora-frozen-transformer-chain-relay","2026-10-03T21:05:16+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"a886afac-eb9a-4666-82a2-03fabc82a29f","Tapered Language Models：Mila\u002FCornell\u002FUdeM 用「锥形 MLP」把 LLM 的容量分配「免费升舱」","tapered-lm-tapered-mlp-free-upgrade","2026-06-28T14:15:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"abd8eb9f-c106-4899-bf24-b6bbed3da5e3","EPFL论文实测:注意力砍维度而非砍token,只算三成点积","hisa-hidden-dimension-sparse-attention-rope","2026-11-20T09:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"2d4e00e8-93d2-4a76-ad68-a362493ea3a0","Google线性注意力Muir:4K训练外推千万token","google-muir-linear-attention","2026-11-07T17:10:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"f3edd5fd-5a5c-447f-8704-500856b96d09","ZHA:Zyphra 用树形索引把注意力成本压成次二次","zyphra-zha-tree-indexed-subquadratic-attention","2026-11-02T05:21:18+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"49972f78-da29-4041-8f1f-c57d9d8ceeac","MIT 一行代码修 MTP:八成梯度噪声被滤掉","soft-mtp-kl-gradient-recovery","2026-10-30T17:12:42+00:00"]