Open-weight teams keep costs down by distilling the reasoning ability that others paid fortunes to RL-train. But a paper posted to arXiv on October 15 flags an overlooked trap: supervising distillation with only the teacher's pre/post-RL endpoint shift can import exactly the wrong signal. The paper, Waypoint-OPD, has since topped the Hugging Face daily paper chart with 665 upvotes.
Endpoint shifts can lie
Quick background. Policy-shift distillation takes the difference between a teacher's log probabilities before and after reinforcement learning and applies it as supervision on trajectories the student itself generates. The problem is that this difference is an endpoint snapshot: it cannot say what actually happened inside the run. The paper's core observation is that identical endpoint shifts can arise from a consistent, direction-stable strengthening — or from an early swing one way and a late swing back that partially cancel each other. To a student these two cases look exactly the same, yet their learning value is completely different.
A directional gate
The fix is direct: pull an intermediate checkpoint from the same RL run as a waypoint, split the total shift into an early and a late component, then compare their directions on prefixes the student actually visits. When the two directions agree, the teacher's preference evolution was coherent, and the endpoint shift passes through as supervision. When they oppose, the shift at that position is likely noise or a transient intermediate state, and it gets suppressed. Cost-wise, the authors report that the whole mechanism adds only one extra frozen-teacher scoring pass compared with endpoint-only transfer — a nearly free price for the filtering effect.
Experiments and reproduction
The setup: teachers are RL-trained Qwen3-4B models; students are Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B; tasks span mathematical and code reasoning. The authors report that across all six domain-student settings, the method achieves the highest domain-average score among the compared distillation methods. Ablations with the Qwen3-8B student show that removing or shuffling the gates reduces mathematical reasoning performance, supporting the claim that directional agreement — not just any gating — carries the effect. For reproduction, the authors released a verl-based implementation on GitHub (hhhbxloveai/WaypointOPD, published as an anonymous implementation), with a default single-node 8-GPU configuration, DAPO-17K as the training set, AIME24 for evaluation, and the waypoint and final teachers mapped to the Step600 and Step1200 checkpoints. Model weights are not included in the repository.
Why it matters
One easily missed detail: the teacher is only 4B while the students run from 8B to 30B. This is not the classic big-teaches-small setup — what gets transferred is the gain that RL produced, grafted onto larger bases. The object of distillation is not knowledge but the policy improvement reinforcement learning induced. A second signal: distillation research is visibly shifting from asking what to learn to asking when not to learn. The gate is essentially denoising the supervision signal, betting only where the teacher's own evolution was self-consistent. For the open-source community the cost arithmetic is equally clear — RL runs are expensive, but intermediate checkpoints are a free byproduct of every run, and one extra scoring pass is negligible.
So the next time you finish an RL run, don't just keep the final checkpoint. That intermediate waypoint may be the most valuable asset you own for distillation.