On the training menu for reasoning models, on-policy distillation (OPD) has spent the past year cast as an alternative to RLVR: where RLVR hands out sparse outcome-level rewards, OPD keeps a teacher model scoring every step the student generates — dense, token-level supervision. It sounds more precise. It is also more expensive, because the big teacher stays in the training loop the whole time.
The teacher's signal is dirtier than you think
The first thing Yi Ding and Ruqi Zhang of Purdue University do in their August 31 arXiv paper (2608.31046) is quantify the quality of that supervision. The finding is awkward: teacher scores carry substantial noise, and the larger the teacher, the higher the noise prevalence. The cause is structural — the teacher is scoring student-generated trajectories that are inherently off-policy for it, grading a distribution it does not normally work in.
Then comes the counter-intuitive layer: the student barely notices. Keep the noisy supervision or strip it out, and the student converges to comparable performance either way.
So where do the gains come from?
The authors kept digging and found learning signals concentrating on low-log-probability tokens. The sharper cut: swap the teacher-provided advantage for a single fixed negative advantage, and performance still matches. Which points to an uncomfortable explanation — most of OPD's benefit comes from the act of suppressing low-probability tokens itself. And that act needs no teacher at all.
OPSA: showing the teacher the door
Following that conclusion, the paper proposes On-Policy Self-Adaptation (OPSA): no supervision signal, entropy-adaptive negative advantages that assign stronger learning signals at high-entropy positions, suppress tail tokens, and redistribute probability mass back across head tokens. Numbers as reported by the paper: on a Qwen3-1.7B base, OPSA lifts Avg@32 on AIME24 by 35.41 points (a 263% relative gain), more than doubles Pass@32 across all three benchmarks, and beats OPD itself by 16.77 Avg@32 points on AIME24. The paper also claims the findings generalize across model families and tasks, across 20 pages and 12 figures.
The paper currently tops the Hugging Face Daily Papers board (86 upvotes at the time of writing) — the community clearly felt the "paying a teacher for nothing" sting.
So what
For training practitioners, the actionable takeaway is not "fire the teacher immediately" — it is "run the ablation first." If most of your OPD gain comes from the suppression effect, your teacher-inference budget is paying two salaries for one person's job. The industry-level metaphor is colder: the brand premium attached to the word "distillation" may far exceed the knowledge actually transferred. Caveats apply — this is a single paper, and AIME24-style math benchmarks do not equal general capability. But the fixed-negative-advantage experiment is one every believer in the distillation narrative should try replicating by hand.