In pretraining there is a class of interventions that only act during a window and then get switched off for good. A temporary weight-decay schedule is the standard example. The intuition is that once the window closes, the story is over. Measured, it is not.

The paper Delayed Effects of Temporary Weight Decay in Muon Pretraining (arXiv:2611.17876, submitted to arXiv on 13 November 2026, with authors from Meta, Mila-Quebec AI Institute, Université de Montréal and Samsung SAIL Montreal) documents a set of counterintuitive curves.

At removal, it is worse

The main configuration trains decoder-only Transformers with Muon at scales from 162M to 3.4B, on FineWeb100B (a 100B-token sample of FineWeb). The method is simple: turn decoupled weight decay on inside a training window and off outside it.

The most striking case is the 1.4B model: coefficient 0.15, applied over the first 44% of training, with the schedule selected at 162M and transferred over without retuning. When the window ended at update 12,320, this run was 0.26 nats worse than the no-decay reference. Both runs then continued with exactly the same update rule, and the treated run slowly clawed its way back, overtaking the reference at update 20,440 (roughly 73% of training) and finishing 0.0275 nats better, and also 0.0264 nats ahead of the best sampled constant-decay baseline. Sampling the best schedule directly at 1.4B improved on the transferred one by only 0.0020 nats.

In other words: the measurement at removal and the outcome at the end have opposite signs.

Three things that get conflated

The paper then separates this effect: a temporary schedule has at least three separable dimensions, yet they are routinely treated as one thing.

Start time flips the sign. At all three scales, a 10% window with the same coefficient that starts at initialization beats both the no-decay and the best-constant references; the same-duration, same-coefficient window that starts at 60% of training loses to both. Because these windows all sit inside the constant-learning-rate phase and apply the same multiplicative shrinkage, this is a timing effect, not a difference in total shrinkage.

Duration matters more than the coefficient. At 162M trained for 16,000 updates (five times the compute-optimal budget), in a 4x3 grid of durations and coefficients, varying duration at fixed coefficient moves the final improvement by 0.026-0.035 nats, while varying the coefficient at fixed duration moves it by only 0.007-0.010 nats. Even with total log-shrinkage D matched at about 2.1, stretching the window from 700 updates to 3,564 more than doubles the magnitude of the final improvement. What decides the outcome is the temporal distribution, not how much was shrunk in total.

Evaluation point rewrites the ranking. At 405M, comparing at a common final checkpoint and comparing after an equal number of post-removal updates give opposite orderings of the early-start windows: the former prefers the 20% start, the latter prefers the start at initialization.

The norm deficit and relative steps

The mechanism lands on weight norms. Early decay shrinks the weights, and the effect survives removal: in the 1.4B run, norms at removal were only 12-16% of their no-decay values, and were still just 60-66% as large at the final checkpoint. Smaller norms mean larger relative steps (the update perpendicular to the weight direction, divided by the weight norm), and that gap converges slowly. The paper's approximate norm-dynamics recurrence runs from update 400 to 2,800, where the recurrence's norm term grows roughly fourfold while the gap between the two runs changes much less, so the relative-step ratio falls only from about 1.6-1.8 to about 1.1.

A fixed-norm replay tests this directly. Holding weight norms at their initial values and replaying only the measured relative-step trajectories, the early-decay trajectory ends 0.022-0.023 nats below the no-decay trajectory, against a 0.0171-nat difference between the two original Muon runs. The trajectory-swap experiment is blunter still: the steps after removal account for 0.0181 of the 0.0221-nat replay difference, whereas replaying only the early-decay trajectory during the intervention and then returning to ordinary Muon buys just 0.0003 nats. The opposite perturbation behaves as predicted too: an early learning-rate boost raises norms and depresses later relative steps, and all three tested boosts finished with higher loss.

Not a weight-decay monopoly

To rule out the idea that this is peculiar to multiplicative weight decay, the paper switches to a qualitatively different intervention, the off-diagonal Gram penalty. At 162M, 10% windows starting at 0%, 20% and 60% change final loss by -0.0147, -0.0080 and +0.0209 nats respectively, the same timing dependence; across five paired 162M runs the early Gram penalty improves final loss by 0.0142 nats relative to no regularization. Decomposed, the radial component nearly reproduces the whole penalty while the tangential component has little effect. What does the work is the norm, not the direction.

Across to 3.4B, and what it does not cover

The 3.4B results point the same way. The 12,320-update version transferred from 1.4B ends at final loss 2.4638 against 2.4832 for no decay, with mean zero-shot accuracy of 66.90 against 65.30 across five tasks; neither constant reference reduces loss (2.4843 and 2.4895), and their mean accuracies are only 65.68 and 65.69. The cost is waiting: at removal this run is 0.2973 nats behind the reference and only settles below it from update 25,400.

The paper draws its own boundaries. Most configurations outside the replicated comparisons use a single seed; the coefficients are not universal; mechanism measurements are concentrated at 162M; at 405M and 1.4B the downstream evaluations show no clear advantage of temporary decay over constant decay; and with AdamW the effect holds at 162M but not at 405M, so it is not optimizer-independent. A 10% linear warmup does not remove the delayed crossover either: in a 16,000-update 162M run, temporary decay is 0.0237 nats worse at update 8,000 but 0.0373 nats better by update 14,400.

So what

The methodological consequence is direct: evaluate a temporary optimization intervention at the training budget where you actually intend to use the model, not at the moment it is removed. Choosing schedules by the loss at removal can pick one that is worse in the end.

Cross-checking of the mechanism is already under way. A separate team reports in arXiv:2608.24814 that loss dynamics are governed primarily by the ratio of learning rate to parameter norm, and describes delayed acceleration as a recurring effect of norm control, which is the same language as the relative-step account here. Muon's own design writeup (https://kellerjordan.github.io/posts/muon/) explains why smaller norms enlarge relative updates: the update matrix is approximately orthogonalized by a Newton-Schulz iteration, so update magnitude is largely controlled, and the smaller the weights, the larger the same-sized update becomes relative to them.

Three directions converge on an uncomfortable conclusion: the curve you see mid-training may not be a preview of the final result at all, just a story about that stretch of training.