Cosine annealing is close to a default answer for learning-rate scheduling in pretraining, and when people tune, they stare at the nominal learning-rate curve. A new paper puts the warning right in its title: Your LR Schedule May Not Be Principled. The problem is not that your curve is insufficiently refined — it is that the object being scheduled was chosen wrong in the first place. The word Principled in the title takes aim at a familiar state of affairs: schedule curves have long been picked by convention and ablation, not designed from a clearly defined scheduling object.
The gains from weight normalization have another source
The backdrop: weight-normalized (WN) training is emerging as an alternative to conventional weight decay (WD), with recent studies reporting substantial pretraining loss gains. The paper (arXiv:2610.40362, by Qifan Yu, Xinyu Ma, Xingyan Bin, and Di He; the Hugging Face paper page lists the team under ByteDance Seed) first cuts into the attribution: those gains are driven primarily by the effective learning rate (ELR) trajectory that WN induces, not by normalization itself. The evidence: once the two ELR trajectories are matched, the training loss dynamics of WN and WD largely collapse onto the same curve.
In other words, the variable that actually matters in the WN-versus-WD debate is the effective learning rate, quietly rewritten by normalization; schedule design should target ELR directly rather than the nominal LR. Once the attribution shifts, both camps' rival narratives wobble: WD users need not envy WN's numbers, and WN users should not credit normalization for them.
AIR: a theoretically derived 1/√t plus an affine adjustment
Following that principle, the paper proposes AIR (Affine Inverse-square-Root decay): a theoretically derived 1/√t decay with an affine adjustment on top, so that over a finite training horizon both the peak and the terminal learning rate land on prescribed values. The authors further identify late-stage ELR scheduling as the key source of the gains — a shared late schedule attenuates the effect of earlier LR choices, so the shape of the endgame curve carries the most weight. This also explains a common piece of folk experience: the exact shape of early warmup often matters less than assumed, because a sufficiently good late-stage ELR dilutes the differences of earlier choices. Tuning priorities invert accordingly — fix the endgame curve first, then work back to the start.
Tested on 128 B-token MoE pretraining
The experiment is not small: 128 B-token Mixture-of-Experts pretraining. Per the paper, AIR outperforms cosine and other candidate schedules on training loss, downstream performance, and hyperparameter robustness; at matched cumulative learning rate, terminal training loss is about 0.01 lower than cosine. More persuasive is the reverse check: applying the same ELR trajectory back to WD training reproduces the corresponding dynamics — the benefit is not exclusive to fixed-norm training, and ELR is the principled object to schedule.
The boundaries are equally clear, though: this is a single team's 128 B-token experiment, the HF paper page has gathered only single-digit upvotes a day after release, and no third-party replication has appeared — too early to declare cosine retired. But "look at ELR before touching the schedule" is a lens worth keeping in every tuner's toolbox.
So what?
What this paper really moves is not one specific schedule curve but the object of tuning itself: the nominal LR is the number in your config file, while ELR is the step size your weights actually feel. Next time a reproduction of someone's recipe fails, before blaming the random seed — plot both sides' effective learning-rate trajectories and compare. That is probably the earlier thing to check.