A strange failure mode is troubling teams running on-policy distillation (OPD): the student model answers correctly, then simply cannot stop. A paper posted to arXiv on September 17 (arXiv:2609.20511, a Microsoft-UNC collaboration, 30 pages) offers a surprising diagnosis — the culprit is usually not the reward, not the data, but a detail almost nobody checks: the teacher's and student's EOS termination tokens disagree.
The symptom: correct answers, then thousands of wasted tokens
The paper opens with a vivid case. One Qwen3 rollout reaches the correct answer at token 1,094, then generates 7,098 redundant tokens; another hits the right result at token 462 and spends 94% of its 8,192-token budget repeating punctuation. The answer is right, the grade is right — and almost the entire generation is waste.
Nor is this a tail of unlucky samples; it is where training converges. Over 200 updates, mean response length climbs into the token budget and the fraction of clipped (never-terminated) responses approaches 100%, under both prompt templates tested.
The root cause: stopping probability lands on an unsampleable token
The authors measure the exact quantity the symptom is about: at the position where the student actually stopped, how much probability it places on stopping. The Qwen student does learn to terminate — stopping probability rises to roughly 0.87 around step 35 — then unlearns it; by step 150 the probability of stopping where it stopped is indistinguishable from zero, exactly when length saturates the budget. Llama and Gemma never get there at all.
Mechanistically: a base student treats one token as terminal, while its post-trained teacher ends assistant turns on a different one — <|endoftext|> versus <|im_end|> for the Qwen pair, one terminal id against three for Llama. The teacher is confident the turn is over, but that confidence sits on a token the base student's decoder can never sample, so it never appears in a rollout and never receives gradient; meanwhile the token the student can emit gets no support from the teacher. Terminating becomes an action the objective cannot reward, and continuing is all that remains.
The fix: treat stopping as one semantic action
Aligning the decoding stopping set alone is insufficient — the paper verifies this explicitly. What works is aggregating the student's and teacher's functionally equivalent EOS tokens into a single semantic stopping action, supervised with the teacher's summed terminal mass (EOS_MODE=semantic_class). Length inflation reverses with it: Gemma drops from saturating the 7,168-token budget at a ~100% clip rate to a mean of roughly 2,000 tokens — close to the teacher's own median on the same prompts — with clipping near zero; Llama comes down to ~3,500-4,000 tokens and roughly halves its clip rate. The finding reproduces across Qwen3, Llama 3.2, and Gemma 3, evaluated on AIME24/25 and AMC23.
What remains unfixed, and the community reaction
The fix is not the whole story: in a 400-update long-horizon run on a K2-Horizon student-teacher pair, a distinct late-stage length inflation persists even after termination alignment — mismatch is an important source, but not the only one. The code is open-sourced (github.com/UNCSciML/opd-eos, including a five-condition EOS ablation), and under the paper page a practitioner commented that the length inflation he had been chasing might be a token-id mismatch rather than reward hacking — "a bunch of us have been debugging the wrong thing entirely."
For teams running OPD, the practical takeaway is a diagnostic order: when length runs out of control, check teacher-student termination-token alignment first (inspect the teacher's EOS distribution at the positions where the student terminates) before blaming the objective, the data, or the horizon. A token-level detail decides most of the token spend and wall-clock time of a distillation run.