One query reaches 71.5% state coverage in on-policy distillation

Distillation practice assumes more training data is better. A paper uploaded to arXiv on Sep 3 — Rethinking On-Policy Distillation II: One Training Example — pushes that assumption to the limit: on-policy distillation trained on a single example keeps improving. Once the numbers are on the table, the marginal value of "collecting more data" needs a second look.

How far one query goes

Background first: on-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. The authors state that existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. The Thinking-Space team (README contacts are tsinghua.edu.cn addresses) took the data-minimal limit literally: train on a single query.

Three results. First, one-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families. Second, the paper defines state coverage — the fraction of the states full-data OPD visits that a query set's rollouts reach — and a single query already hits 71.5%, with most of it within the first 100 steps. Third, adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach 98.9% and match full-data training on roughly 17k queries.

Data-overfed, algorithm-starved

The sharper finding is absorption rate: alignment with the teacher slows at a similar pace whether OPD trains on one query or all 17k, and even a fixed set of states takes hundreds of steps to absorb. The paper's one-line verdict: OPD is data-overfed but algorithm-starved — rollouts quickly expose broad supervision, while the student absorbs it increasingly slowly.

Two extensions. The state-coverage result carries over to multi-teacher MOPD, where 16 semantically diverse queries per domain match full-data MOPD. And in stress tests, content-light templates and off-domain WildChat queries also approach the real-query baseline: task content and induced state coverage can come apart. The repo even ships a template mode where the student writes its own training input.

Open source and reproduction

Code is on GitHub (Thinking-Space/One-Shot-OPD, Apache 2.0), implemented by extending veRL; reference numbers come from a single 8-GPU node with H100/A100 80GB. Evaluation spans four domains: math (MATH-500, AMC 2023, AIME 2025, avg@16), code (LiveCodeBench v6, avg@3), instruction following (Multi-IF across eight languages), and tool use (BFCL v3, avg@8). This is Part II of a series; Part I went up in April, and the team hopes these findings direct future work toward the step efficiency of OPD.

So what

For post-training teams the lesson is not "distill on one example from now on" but a reallocation of effort: the marginal return on data engineering is lower than assumed, and how fast a student absorbs supervision is the next battleground. Read alongside Purdue's early-September finding that fixed negative advantages can match the teacher, the question "what is OPD actually learning" is getting re-examined from several directions at once. Paper and code: arxiv.org/abs/2609.04172, github.com/Thinking-Space/One-Shot-OPD.