On-policy distillation (OPD) has become a popular way to accelerate post-training: a frozen teacher provides dense token-level supervision on the student's own rollouts, far more efficient than sparse outcome-only signals. But a paper submitted to arXiv on Sep 2 and trending on Hugging Face Daily Papers on Sep 8 points out a widely ignored blind spot: vanilla OPD applies teacher supervision uniformly across prompts, without ever checking whether the teacher is actually reliable on the prompt at hand.
A Confidently Wrong Teacher Is Worse Than No Teacher
The core observation rests on a mathematical property: OPD typically uses reverse KL as its objective, and reverse KL is mode-seeking — a confidently wrong teacher induces a strong yet misleading update. Distributional proxies such as entropy or teacher-student likelihood agreement measure uncertainty or agreement, but they do not directly verify outcome correctness. They can catch an uncertain teacher; they cannot catch one that is confidently wrong.
How the Prompt-Level Gate Works
TGOPD (Teacher-Gated On-Policy Distillation) compresses its recipe into four words: verify before you distill. It estimates teacher reliability on the current prompt from a small set of verifier-scored teacher probes. Prompts that pass the check go to dense OPD; prompts that fail are routed to verifier-grounded GRPO instead — the two paths are mutually exclusive. Teacher trust turns from a default assumption into an explicit checkpoint.
Gains on Two Levels
The experiments cover 4B and 35B students across mathematics, code, and instruction following. The paper reports that TGOPD outperforms Vanilla OPD in all six single-domain settings, and achieves higher seven-benchmark averages at both scales under multi-domain training. The more practical number is about compute: in asynchronous OPD the teacher node sits idle much of the time, and packing reliability estimation into that otherwise-idle capacity raised teacher-node GPU utilization from 9.8% to 78.9% in the measured 4B single-domain run — the gate is essentially free.
So What
The takeaway is not another training trick but an engineering principle: dense supervision should be admitted on verification. The denser the signal, the more you should ask whether its source is reliable — true for distillation, equally true for reward models in RLHF and tool feedback in agent workflows. Routing supervision by trustworthiness (distill when reliable, reinforce when not) is a pattern likely to reappear across post-training pipelines. The paper is at arXiv:2609.02998.