[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-kakao-moe-hyperparameter-transfer-mup":3,"news-related-4a913e73-3626-4339-99d8-de5e12312071":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"4a913e73-3626-4339-99d8-de5e12312071","Kakao 新论文:小模型代理训练算准 155B MoE 的 10T token 学习率","COLM 2026 论文提出两步超参迁移框架:μP 让最优学习率跨模型宽度可迁移,再沿 token 维度做 log-log 线性回归外推(R²=0.95),预测出 3.85×10⁻⁴ 的学习率,支撑 155B\u002F17B MoE 从零预训练 10T token 全程无 loss 尖峰。","训练一个千亿参数的 MoE 大模型,最贵的一步可能不是买 GPU,而是猜学习率。韩国 Kakao 的研究团队 8 月 20 日在 arXiv 发表论文(已被 COLM 2026 接收),提出一套两步超参迁移框架,核心主张是:用一个 10.8B 的小模型跑几组短程代理实验,就能直接算出 155B 参数 MoE 模型预训练 10 万亿 token 该用的学习率,全程不需要在目标规模上做任何扫参([arXiv:2608.20061](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.20061))。\n\n## 问题:扫参在万亿 token 尺度上是个无底洞\n\nMoE 架构让模型容量扩展不再伴随计算成本的同比例上涨,DeepSeek-V3、Qwen3-235B-A22B、Kimi-K2.5、GLM-5 等主流开源模型全部采用这一设计。但容量换来的代价是超参空间的膨胀:专家路由、负载均衡引入的新自由度,让 MoE 的调参复杂度和成本都显著高于稠密模型。其中学习率尤其麻烦——它的最优值对模型规模和 token 预算都高度敏感,这意味着每次换规模都要重新扫一遍。论文算了一笔账:如果沿用传统的模型规模 × token 规模二维扫参,仅模型规模那一维(比代理模型宽 1.5 倍、2 倍)就要额外烧掉 240.3 ZFLOPs,而代理实验本身只需要 64.8 ZFLOPs。在 10T token 的目标尺度上,这个成本没有实验室扫得起。\n\n## 方法:μP 管宽度,scaling law 管 token\n\n框架拆成两步,分别对付两个维度。\n\n**第一步对付模型宽度。** 团队把 Maximal Update Parameterization(μP)适配到采用多头潜在注意力(MLA)和 Muon 优化器的 MoE 架构上。μP 的价值在于零样本迁移:小模型上找到的最优学习率,可以不加修改地搬到按宽度放大的大模型上。针对 MoE 的参数形状,论文把 router 和专家 FC1 权重划入 matrix-like 参数(受初始化和学习率双重缩放),专家 FC2 权重划入 vector-like(仅初始化缩放);同时固定激活专家数和 MoE 中间维度,只增总专家数和隐藏维度——这是超大规模 MoE 的实际扩展路径。实验验证了关键前提:同一 token 尺度下,代理模型和宽度放大模型的验证损失抛物线曲率与顶点位置高度一致,顶点(即最优学习率)跨宽度可靠迁移。\n\n**第二步对付 token 预算。** 在 10.8B 总参\u002F3.3B 激活(目标模型的 1\u002F4 宽度)的代理模型上训练约 500B token,每 10B token 间隔估一次最优学习率,再用 log-log 空间的线性回归拟合「最优学习率随 token 预算下降」的趋势。拟合优度 R²=0.95,外推出 10T token 尺度的理想学习率为 3.85×10⁻⁴。传统二维扫参在这里被压缩成沿 token 维的一维搜索,模型规模维被 μP 直接消掉了。\n\n## 结果:10T token 训练全程无 loss 尖峰\n\n用预测出的学习率,团队从零预训练了一个 155B 总参\u002F17B 激活的 MoE 基础模型,10 万亿 token 的全程训练损失保持高度稳定,没有出现任何 loss 尖峰。论文强调,目标训练的总计算量约为代理实验总量的 98 倍——换句话说,代理实验的花费不足正式训练的 2%。评测覆盖 MMLU、MMLU-Pro、BBH、MATH、GSM8K、MBPP、HumanEval 及 Global-MMLU(韩\u002F日\u002F越\u002F中)四个方向;与 dots.llm1、GLM-4.5-Air、Hunyuan-A13B、DeepSeek-V4-Flash 等同量级开源 MoE 基座相比,该模型位于 Pareto 前沿:在相近或更低的估算训练算力下,MMLU-Pro 准确率高于 dots.llm1 和 GLM-4.5-Air。作者也坦承局限:未做目标规模的全量扫参来验证预测学习率的最优性(算力上不可行),且宽度与稀疏度是联合扩展的,稀疏维度本身的独立影响无法剥离。\n\n## 所以呢\n\n这篇论文的看点不在模型本身,而在方法学:当训练预算迈过 10T token、参数迈过千亿,「扫参」这个深度学习时代的默认动作正在变得物理上不可能。μP + token 维 scaling law 的组合给出了一条替代路径——把搜索成本从目标规模转移到代理规模,让「一次训对」取代「多次试错」。对于算力紧张、却要啃万亿 token 配方的团队,这套框架值得关注;至于它能否推广到其他 MoE 结构和其他优化器,作者留给了未来工作。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.20061","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"62a0408c-027e-4515-807d-5f19dc5e1390","korean",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"d11f0044-8aef-487c-bebe-89ce4683a4a3","moe",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"b08e7e8a-c528-476b-b2dc-d56056be16c5","en","Kakao paper computes optimal LR for 155B MoE via small proxy runs","Kakao's COLM 2026 paper: muP width transfer + a log-log scaling law (R^2=0.95) predict LR 3.85e-4 for 10T-token pretraining of a 155B\u002F17B MoE.","Training a 100B-parameter MoE model may hinge less on buying GPUs than on guessing the learning rate right. On August 20, researchers at Korea's Kakao published a paper on arXiv (accepted at COLM 2026) proposing a two-step hyperparameter transfer framework. The core claim: run a few short proxy experiments on a 10.8B model, and you can directly compute the learning rate for pretraining a 155B-parameter MoE over 10 trillion tokens — with no sweeps at target scale whatsoever ([arXiv:2608.20061](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.20061)).\n\n## The problem: sweeps are a bottomless pit at trillion-token scale\n\nMoE architectures expand model capacity without a proportional increase in compute. DeepSeek-V3, Qwen3-235B-A22B, Kimi-K2.5, and GLM-5 all adopt this design. But the capacity comes at a cost: expert routing and load balancing add new degrees of freedom, making MoE tuning considerably more complex and expensive than dense models. The learning rate is the worst offender — its optimal value is highly sensitive to both model size and token budget, so every change in scale demands a fresh sweep. The paper does the math: under the conventional 2D sweep across model scale and token scale, the model-scale dimension alone (1.5x and 2x wider than the proxy) burns an additional 240.3 ZFLOPs, while the proxy runs themselves cost just 64.8 ZFLOPs. At a 10T-token target, no lab can afford this.\n\n## The method: muP handles width, a scaling law handles tokens\n\nThe framework decouples into two steps, one per dimension.\n\n**Step one tackles model width.** The team adapted Maximal Update Parameterization (muP) to MoE architectures using Multi-head Latent Attention (MLA) and the Muon optimizer. muP enables zero-shot transfer: the optimal learning rate found on a small model can be applied unchanged to width-scaled larger models. For MoE parameter shapes, the paper classifies router and expert FC1 weights as matrix-like (subject to both initialization and learning rate scaling) and expert FC2 weights as vector-like (initialization only). The scaling path fixes the number of active experts and the MoE intermediate dimension while increasing total experts and hidden width — the practical route for extreme-scale MoEs. Experiments confirmed the key premise: at each token scale, the validation-loss parabolas of proxy and width-scaled models show highly consistent curvature and vertex locations, so the vertex (the optimal learning rate) transfers reliably across widths.\n\n**Step two tackles the token budget.** On a 10.8B-total\u002F3.3B-active proxy (one-quarter the width of the target), the team trained roughly 500B tokens, estimated the optimal learning rate every 10B tokens, then fit a linear regression in log-log space to the downward trend of optimal LR versus token budget. The fit achieves R^2=0.95, extrapolating to an ideal learning rate of 3.85e-4 at the 10T-token scale. The conventional 2D sweep collapses into a 1D search along the token dimension — the model-scale dimension is eliminated by muP entirely.\n\n## The result: 10T tokens with zero loss spikes\n\nUsing the predicted learning rate, the team pretrained a 155B-total\u002F17B-active MoE foundation model from scratch. Training loss over the full 10 trillion tokens remained highly stable, with no loss spikes. The paper notes the target run costs approximately 98x the total compute of all proxy runs — the proxy experiments amount to under 2% of the formal training. Evaluation spans MMLU, MMLU-Pro, BBH, MATH, GSM8K, MBPP, HumanEval, and Global-MMLU (Korean, Japanese, Vietnamese, Chinese). Against comparable open-weight MoE bases — dots.llm1, GLM-4.5-Air, Hunyuan-A13B, and DeepSeek-V4-Flash — the model sits on the Pareto frontier, achieving higher MMLU-Pro accuracy than dots.llm1 and GLM-4.5-Air at comparable or lower estimated training compute. The authors are candid about limits: no full-scale sweep was run to verify the predicted LR's optimality (computationally infeasible), and width and sparsity were scaled jointly, so the isolated effect of the sparsity axis cannot be disentangled.\n\n## So what\n\nThe paper's significance lies not in the model itself but in the methodology. As training budgets cross 10T tokens and parameter counts cross 100B, sweeping — the default move of the deep learning era — is becoming physically impossible. The muP + token-axis scaling law combination offers an alternative: shift the search cost from target scale to proxy scale, and let \"train it right once\" replace \"trial and error many times.\" For teams with tight compute budgets tackling trillion-token recipes, this framework deserves attention. Whether it generalizes to other MoE structures and optimizers is left to future work.","kakao-moe-hyperparameter-transfer-mup","2026-08-24T13:20:00Z","2026-08-24T13:12:49.172333Z","2026-08-24T13:12:49.172342Z",true,"agent",33,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"7958a2f1-028c-4b4e-b134-0d5de9afc1c1","Motif 3 收官:韩国 314B MoE 改用 MIT 许可,从零起步架构首次面向商用","motif-3-mit-license-sovereign-ai","2026-08-24T00:00:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"491f4904-c854-4925-b3e3-e34b8afd5e50","KDA+MLA 混合栈下沉到 1.3B 激活:Ling-3.0-tiny 把 MoE 端侧化,INT4 跑出 115 tok\u002Fs","ling-3-tiny-kda-mla-edge-deployment","2026-08-18T00:00:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"804ab59a-a8d6-4b61-bf74-8f6f2bdae83c","智谱把 Flash 做成一件正经事:一次说清 GLM-5.3-Flash 的架构和 benchmark 真相","glm-5-3-flash-hybrid-attention-architecture","2026-08-27T08:00:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"68072ee1-fc37-4064-ab18-09550ae72d1b","GLM-5.3-Flash 把 320B MoE 跑在国产芯片上:Flash 价位和 $0.15 API 的混合注意力栈","glm-5-3-flash-chinese-chips-hybrid-attention","2026-08-27T03:00:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"e75069c6-f15c-4ff9-8b11-404d705442e8","Upstage Solar Pro 4:把「agent 跑得稳」做成新一代闭源模型卖点","upstage-solar-pro-4-agent-reliability-closed-llm","2026-08-25T03:00:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"6fa1bc74-c98e-476b-bc4c-9ae057105ffb","ParaTempo:免训练并行推理,延迟最高降 32%、token 省三成","paratempo-temporal-confidence-parallel-reasoning","2026-08-24T17:20:00+00:00"]