For the past two years, "on-policy" has been the default answer in LLM post-training: train on the model's own rollouts, and you supposedly get less catastrophic forgetting, sparser parameter updates, and better generalisation. A controlled study from the van der Schaar Lab at the University of Cambridge — submitted to arXiv on September 28 by co-first authors Julianna Piskorz and Antonin Berthon — puts that narrative to a proper test, and the verdict is unkind: in distillation settings, rollout policy is not necessarily the protagonist.

A controlled experiment with three separate knobs

Previous comparisons between supervised fine-tuning and reinforcement learning varied many factors at once, making it hard to isolate what rollout policy actually contributes. This paper decomposes the distillation configuration into three independent knobs — rollout policy (student-generated or teacher-generated), token-level KL direction (forward or reverse), and learning rate — and varies each one separately across the Llama 3 and Qwen2.5 model families, deliberately breaking the conventional pairing of forward KL with off-policy learning and reverse KL with on-policy learning.

The main experiments distil Llama-3.1-8B teachers into Llama-3.2-1B students, with an appendix replicating everything using Qwen2.5-7B-Instruct teachers and Qwen2.5-1.5B-Instruct students. Tasks span three reasoning domains: science, medical (MedReason), and arithmetic (Countdown). Teacher baselines are disclosed too: the Llama-3.1-8B teacher reaches 85.5% accuracy on MedReason, while the Qwen2.5-7B teacher hits 95.5% on Countdown-3.

The real deciding factors are different ones

The core finding fits in one sentence: token-level KL direction more clearly determines task performance and output coverage, learning rate governs catastrophic forgetting and update sparsity, and rollout policy does not necessarily play a central role. Broken down:

  • Forward KL is remarkably robust to rollout policy — performance stays stable and strong no matter who generates the data. The paper backs this with gradient analysis (Theorem C.1): the expected squared norm of the forward-KL gradient has an upper bound independent of the rollout policy.
  • Reverse KL is substantially more sensitive, and clearly favours student-generated rollouts.
  • The authors state plainly: there is no evidence that on-policy rollouts intrinsically preserve prior capabilities or induce sparser updates.

When on-policy still helps

The paper does not dismiss on-policy entirely. On harder variants of the Countdown arithmetic task, on-policy data consistently improves generalisation under both KL directions — although this advantage does not reliably persist after subsequent RLVR training. Another fine-grained finding: reverse KL combined with on-policy rollouts reduces incidental teacher-style transfer, which the authors verified with a dedicated experiment using a teacher induced to reason in Spanish.

So what

The direct implication for engineering teams: if your distillation budget is limited, spend it first on choosing the right KL objective and tuning the learning rate, not on rolling out an expensive on-policy pipeline by default — on-policy means the teacher model runs inference throughout training. The paper's own words encourage practitioners "to weight the additional cost of on-policy distillation over cheaper alternatives" on the same scale. It reframes the question from "on-policy or not" to "under what objective, evaluation setting, and hyperparameter regime does on-policy actually pay for itself".

Paper: On-Policy or Off-Policy Learning?. Two days after release, it has gathered 80 upvotes on Hugging Face Daily Papers.