Fine-tuning a model to follow instructions, or training it further on math, reliably costs it something on the tasks it used to do well. The industry calls this forgetting. But what is forgetting, mechanically? Is a whole body of knowledge being overwritten, or is it merely that the output layer has shifted its preference toward certain tokens?

The question sounds academic and is actually operational: if forgetting were mostly the second thing, you would not need to retrain anything. You could add a fixed bonus or penalty per token at inference time and claw the lost capability back.

A paper posted on arXiv on October 30 (Less Loss, Fixed Weights: Logit Corrections for Forgetting) takes that question seriously, and the answer is discouraging: you cannot claw it back.

Fourteen model pairs, one million tokens of web text

The author works with 14 released base models and their derived counterparts, drawn from three families — OLMo-2, Qwen2.5 and SmolLM2 — at parameter scales from 1B to 7B. SFT, DPO and Instruct rows cover the post-training regimes, the Math and Coder rows cover continued pretraining, and one row compares an intermediate and a final pretraining checkpoint. The scoring text is one million tokens of web text taken from a single DCLM shard of the OLMo-2 pretraining mix, with documents truncated to 2,047 tokens. The leading 90% of the FIT split is used to fit the corrections, the last 10% to select them, and EVAL is held out entirely.

One methodological detail is worth flagging. If you tailor an adjustment to a specific text, you can make any model look better, including the original base model. So the author fits the same kind of correction separately to the base model and to the derived model, and reports only the extra benefit the derived model gets on top of the base. That quantity is called the symmetric recovered share.

Bias alone recovers 9.8%; adding temperature reaches 35.9%

Start with bias alone, restricted to the 4,096 most frequent tokens. Across the 14 pairs the paper reports, the highest point estimate is 9.8%, for Qwen2.5 7B to Instruct, with a 95% interval of 9.2%–10.4%; every other pair sits below 7%. The author states the bound even more bluntly in the repository README: these fitted corrections stay below 0.20 on all 14 pairs.

Adding temperature changes the picture materially. OLMo-2 7B to Instruct recovers 35.9%, its DPO checkpoint 33.5%, and Qwen2.5 7B to Instruct 28.8%. But the scope matters here. The uncorrected loss gap for Qwen2.5 7B to Instruct is only 0.050 nats/token, so there was never much to lose to begin with, while SmolLM2 and Qwen2.5 1.5B to Instruct both stay below 6%.

There is a useful by-product. The fitted temperature lands between 1.08 and 1.28 for the post-trained OLMo-2 models and Qwen2.5 7B Instruct, against 0.996 to 1.013 for the six base models. The author is careful to note that this temperature difference does not separate a pure temperature shift from other logit changes.

The models that lost the most recover the least

The continued-pretraining rows are the ones that sting. Qwen2.5 Math loses 0.485 nats/token at 7B and 0.796 nats/token at 1.5B, the two largest gaps among all the pairs, yet the symmetric recovered share of bias plus temperature is only 1.3% and 1.2%. The Coder group loses about 0.13 nats/token, with recovery shares hugging zero (0.003 at 7B, and -0.001 at 1.5B).

The README puts it most sharply: continued pretraining on math forgets a lot and recovers almost nothing. The author adds that every out-of-sample correction on those math models recovers at most 0.031, and the in-sample one-step closed form at most 0.055.

The most telling control: a randomly damaged model also "recovers" 29%

This is where the paper earns its keep. Rather than just reporting the numbers it measured, the author runs synthetic experiments with known answers. Random noise is added to the output weights (Gaussian noise with standard deviation 0.25, against original weights with standard deviation 0.5), with no global bias planted and no temperature change — pure random damage.

Then the same bias-plus-temperature correction is applied. The symmetric recovered share comes out at 0.290, and 0.076 for bias alone.

In other words: on a model where there is nothing in principle to recover, this correction still recovers roughly three tenths. The paper draws the conclusion plainly — recovery alone does not identify the training mechanism.

The author also runs the opposite control. With a planted bias, recovery exceeds 90%; with doubled logits the derived-only recovery exceeds 90% and the fitted temperature is 1.987, which shows the measurement pipeline itself is sound. A further proxy case keeps the weight noise but shifts the data-generating prior, so the base model gains too; subtracting that gain drops recovery from 0.474 to 0.299.

So what

The conclusion is not that forgetting is unfixable. It is the more precise claim that most of the real-model loss increase stays out of reach of these restricted fits. In the section on what is not claimed, the author is blunt: one web shard, three model families, models at 7B or below, and a 14-pair comparison that is exploratory in nature. Prior shift onto excluded tokens is not measured, and no temperature-only baseline was fitted. The author also discloses that the scoring text is probably training data for OLMo-2. The original sweep took about 13.5 A100-hours.

But the paper leaves behind one reminder that is directly useful to a reader. Any claim of the form "method X recovers Y% of forgetting" has to answer a question first: if you ran this method on a randomly damaged model, what would it report? This paper happens to have that number already computed: 29%.

The paper and its data are public. Code and per-document loss sums sit in a GitHub repository, with a DOI-backed archive on Zenodo, everything CC BY 4.0. The repository currently shows 4 commits, 0 stars, and no third-party replication, so every number above should be read as something the author measured rather than as settled industry consensus.

Repairing forgetting properly means going back into the weights. Clawing a little back at the output layer tops out around thirty percent — and you had better compare against a randomly damaged control before you believe it.