Training a 100B-parameter MoE model may hinge less on buying GPUs than on guessing the learning rate right. On August 20, researchers at Korea's Kakao published a paper on arXiv (accepted at COLM 2026) proposing a two-step hyperparameter transfer framework. The core claim: run a few short proxy experiments on a 10.8B model, and you can directly compute the learning rate for pretraining a 155B-parameter MoE over 10 trillion tokens — with no sweeps at target scale whatsoever (arXiv:2608.20061).

The problem: sweeps are a bottomless pit at trillion-token scale

MoE architectures expand model capacity without a proportional increase in compute. DeepSeek-V3, Qwen3-235B-A22B, Kimi-K2.5, and GLM-5 all adopt this design. But the capacity comes at a cost: expert routing and load balancing add new degrees of freedom, making MoE tuning considerably more complex and expensive than dense models. The learning rate is the worst offender — its optimal value is highly sensitive to both model size and token budget, so every change in scale demands a fresh sweep. The paper does the math: under the conventional 2D sweep across model scale and token scale, the model-scale dimension alone (1.5x and 2x wider than the proxy) burns an additional 240.3 ZFLOPs, while the proxy runs themselves cost just 64.8 ZFLOPs. At a 10T-token target, no lab can afford this.

The method: muP handles width, a scaling law handles tokens

The framework decouples into two steps, one per dimension.

Step one tackles model width. The team adapted Maximal Update Parameterization (muP) to MoE architectures using Multi-head Latent Attention (MLA) and the Muon optimizer. muP enables zero-shot transfer: the optimal learning rate found on a small model can be applied unchanged to width-scaled larger models. For MoE parameter shapes, the paper classifies router and expert FC1 weights as matrix-like (subject to both initialization and learning rate scaling) and expert FC2 weights as vector-like (initialization only). The scaling path fixes the number of active experts and the MoE intermediate dimension while increasing total experts and hidden width — the practical route for extreme-scale MoEs. Experiments confirmed the key premise: at each token scale, the validation-loss parabolas of proxy and width-scaled models show highly consistent curvature and vertex locations, so the vertex (the optimal learning rate) transfers reliably across widths.

Step two tackles the token budget. On a 10.8B-total/3.3B-active proxy (one-quarter the width of the target), the team trained roughly 500B tokens, estimated the optimal learning rate every 10B tokens, then fit a linear regression in log-log space to the downward trend of optimal LR versus token budget. The fit achieves R^2=0.95, extrapolating to an ideal learning rate of 3.85e-4 at the 10T-token scale. The conventional 2D sweep collapses into a 1D search along the token dimension — the model-scale dimension is eliminated by muP entirely.

The result: 10T tokens with zero loss spikes

Using the predicted learning rate, the team pretrained a 155B-total/17B-active MoE foundation model from scratch. Training loss over the full 10 trillion tokens remained highly stable, with no loss spikes. The paper notes the target run costs approximately 98x the total compute of all proxy runs — the proxy experiments amount to under 2% of the formal training. Evaluation spans MMLU, MMLU-Pro, BBH, MATH, GSM8K, MBPP, HumanEval, and Global-MMLU (Korean, Japanese, Vietnamese, Chinese). Against comparable open-weight MoE bases — dots.llm1, GLM-4.5-Air, Hunyuan-A13B, and DeepSeek-V4-Flash — the model sits on the Pareto frontier, achieving higher MMLU-Pro accuracy than dots.llm1 and GLM-4.5-Air at comparable or lower estimated training compute. The authors are candid about limits: no full-scale sweep was run to verify the predicted LR's optimality (computationally infeasible), and width and sparsity were scaled jointly, so the isolated effect of the sparsity axis cannot be disentangled.

So what

The paper's significance lies not in the model itself but in the methodology. As training budgets cross 10T tokens and parameter counts cross 100B, sweeping — the default move of the deep learning era — is becoming physically impossible. The muP + token-axis scaling law combination offers an alternative: shift the search cost from target scale to proxy scale, and let "train it right once" replace "trial and error many times." For teams with tight compute budgets tackling trillion-token recipes, this framework deserves attention. Whether it generalizes to other MoE structures and optimizers is left to future work.