Vanilla SFT has a side effect nobody likes to advertise: the model learns to follow instructions and forgets what it knew. The measurements in this paper are blunt about it. After instruction tuning, the Qwen3 base models score worse than before fine-tuning on 6 or 7 tasks out of 8; instruction following jumps (IFEval climbs from 45.5 to 76.3 at the 4B scale), but that gain is bought with across-the-board regressions in math, code and knowledge.
The mechanism: freeze the pretrained self as a teacher
A team from MIT, Arizona State and UT Austin proposes Ambient SFT. The idea fits in one sentence: add a Temperature Matching term to the standard SFT loss, so the fine-tuning model must match the high-temperature conditional distributions of a frozen copy of its own pretrained model. Temperature is the trust knob. As T goes to infinity the matching term vanishes and you are back at vanilla SFT; at T = 1 the term is exactly the classic knowledge distillation loss. All experiments use T = 2.5; each step takes the plain SFT loss with probability 0.75 and the matching loss otherwise; teacher and student are the same model, and training runs about 1.1x slower for the same number of steps. The 1/T-squared weighting that ships with Hinton-style distillation was tried and, in the authors' experiments, is not optimal across all settings. The paper is candid about positioning: this is a recipe for data-limited settings, not compute-limited ones.
The theory side offers three guarantees: a target distribution can be recovered from high-temperature conditionals alone; the conditionals of any two models contract as temperature grows; and under a log-linear model assumption, Ambient SFT strictly beats both vanilla SFT and distillation. The name continues the first author's Ambient Diffusion line, which trains diffusion models from corrupted data. Here the small fine-tuning set is treated as the corruption, and the base model supplies denoised supervision. The mechanism check runs on GPT-2 124M over OpenWebText, about 9B tokens: the model learns from its own synthetic data sampled at different temperatures, and even at high corruption strength the validation perplexity on clean data stays low.
The numbers: three regimes tested
For instruction tuning, 1% of Dolci-Instruct-SFT (about 21K examples) is enough. Qwen3-1.7B improves 20.3% on average relative to vanilla SFT across eight benchmarks, with MATH going from 24.3 to 40.4; the 4B, 8B and 14B models gain +8.0%, +10.2% and +8.7%. The effect transfers across families: OLMo-3-7B +15.4%, Llama-3.1-8B +8.3%. In the paper's own accounting, Ambient SFT at 8B matches vanilla SFT at 14B on the eight-benchmark average, and on MATH alone Ambient SFT at 1.7B already exceeds vanilla SFT at 8B.
Against other regularizers on the same 1.7B setup: L2-SP +15.3%, KD at T = 1 +17.0%, Ambient SFT +20.3%, while NEFTune and label smoothing land near zero. The reasoning regime separates them further. Trained on 1% of Dolci-Think-SFT, OLMo-3-7B with Ambient SFT beats vanilla SFT on 5 of 6 benchmarks (it trails GSM8K by 1.2 points) for an average relative +3.5%; KD and L2-SP come in at -7.3% and -37.1%, which is why the paper calls Ambient SFT the one regularizer in that comparison that still improves reasoning scores, and reports it is also more robust to prompt variations. In domain adaptation, the News validation loss drops from 2.01 to 1.68. The data-scaling curve is worth remembering: +15.4% average at 1% of the data, +3.4% at 10%, and still +8.6% at full scale. Evaluations run on the Ai2 olmes suite with a vLLM 0.13.0 backend.
The cold water
First, status: this is arXiv v1, marked Work in progress by the authors on the abstract page, and not yet peer reviewed. Second, IFEval is the one benchmark where Ambient SFT loses to vanilla SFT (52.3 versus 58.8 at 1.7B), and instruction following is precisely what SFT data is for; the fix the paper offers is a per-source temperature across the 21 sources of the mixture, which pulls IFEval back to 58.2 and lifts the average to +21.1%, while also confirming that a single temperature has a blind spot. Third, the boundary: gains thin out with scale (+20.3% at 1.7B down to +8.7% at 14B) and with data (+3.4% at the 10% setting). Fourth, lineage: the paper itself places the method on the extension of knowledge distillation and the LwF line of continual learning. In classic distillation the teacher is stronger; here the frozen base model is strictly weaker at the target task and only provides coarse supervision, and temperature, a fixed constant in LwF, becomes an explicit trust knob. The 2024 self-distillation route differs too: it runs SFT on data generated by the model, while Ambient SFT matches the teacher's high-temperature distribution directly. Also, no code is released with the paper, so do not assume an open-source drop.
So, what is the takeaway? When your fine-tuning data is scarce, do not run vanilla SFT naked. Let the pretrained model teach itself at high temperature; one extra loss term cheaply buys back old capabilities. But the paper is equally clear about the window: the less data and the smaller the model, the more this pays. How much fine-tuning data you have decides whether this is medicine for your pipeline or a footnote.