Feed a language model 100 tasks in a row, and by the time it finishes the last one, how much of what it learned survives? The baseline answer from a Johns Hopkins team is brutal: 1.2%. The paper, submitted to arXiv on Sep 7 and now #1 on Hugging Face's Sep 16 Daily Papers with 287 upvotes, targets the old nemesis of fine-tuning — catastrophic forgetting — and reaches a counterintuitive verdict: stop agonizing over which anti-forgetting mechanism to pick. Compose them.

Quantifying Forgetfulness First

The paper defines "long-horizon memorization": a model learns 100 query-answer tasks through continual supervised fine-tuning, without keeping earlier training examples and without task identifiers at inference. Under this ruler, naive sequential fine-tuning ends with an average final retention of just 1.2% — nearly everything gets washed away by subsequent updates.

To keep conclusions from being data-specific, the team built three 100-task datasets: Symbol-QA maps random six-character keys to four-character values for pure symbolic memory; LLM-QA consists of LLM-generated fictional facts; Real-QA is filtered and remixed from ten public QA sources, closer to real-world knowledge.

How the Combination Works

The core hypothesis: mechanisms addressing complementary sources of forgetting work better when composed. Compositions run along two design dimensions:

  • Anchors decide what to preserve: generative replay for the data anchor, self-distillation against the previous model state for the function anchor, SI or online EWC for the weight anchor.
  • Low-rank allocation rules decide where updates live: shared LoRA, merged LoRA, O-LoRA, and a sequential adaptation of OSRM.

The design space explodes combinatorially, so the team first ran task-level successive halving (a 90→45→23→10 funnel), then a 2^4 factorial experiment to measure individual and interaction effects. The winning recipe, si_sd_replay_merge — all three anchors plus merged LoRA — ranks among the top 3 on all three datasets and lifts average final retention from 1.2% to 34.9%, which the paper calls a 28-fold improvement. The data anchor and merged LoRA contribute the largest gains and interact super-additively on all three datasets.

Is 34.9% the Cure?

No — but this may be the first time the methodology of this whole track has been laid out so systematically. Two things stand out. First, 34.9% still means roughly two-thirds of the knowledge is lost; long-horizon memory is far from solved. Second, the engineering trade-offs are now explicit: O-LoRA and OSRM state grows linearly with the number of tasks, while shared/merged LoRA keeps learner state constant — friendlier for production models that keep rolling forward. The backbone is Qwen3-4B-Base, and the paper also evaluates general capability on GSM8K, MATH, MGSM, and MMLU-Redux after task 100. Code and all three datasets are open-sourced on GitHub.

So What

For teams doing continual domain fine-tuning, the value here is not the 34.9% number but the methodology — mechanisms are composable and their interactions are measurable. Rather than endlessly tuning a single anti-forgetting trick, stack protection across data, function, and weight layers. As models start to behave like employees who keep receiving on-the-job training, memory retention turns from an academic metric into a cost metric: every task forgotten is real money spent retraining.

Paper: arXiv:2609.06986