Is distillation or reinforcement learning the cheaper post-training path? Teams have argued about this for a year, usually with GPU-hour counts or gradient-step ratios—rarely with a matched training-FLOPs comparison. A new paper from Salesforce AI Research closes that gap: it puts on-policy distillation (OPD) and GRPO into one FLOPs accounting framework and sweeps task difficulty, student/teacher sizes, training domains, and optimization choices. The findings are counter-intuitive on several fronts.
External teachers: the intuition collapses
OPD's pitch is straightforward: the student trains on its own rollouts while a frozen teacher scores every token. Dense per-token supervision sounds far stronger than GRPO's outcome-only reward—especially on hard problems, where a student that rarely solves correctly leaves GRPO with almost no positive signal, while the teacher can still supervise step by step.
The measurements punch through that intuition. The team stratified DAPO-Math-17K by mean@32 solve rates of both student and teacher into eight subsets and retrained: in the cell most favorable to OPD on paper (student near-zero, teacher above 0.6), GRPO still dominated in-domain at lower FLOPs. None of nine ablation axes—student initialization, learning rate, batch size, KL estimator, thinking mode, student size, training domain, RL framework—flipped the ordering. The gap on code is starker: GRPO hit 46.4% on LiveCodeBench v6 at 0.6 EFLOP, while OPD with a 32B external teacher peaked at 28.7% after burning 68.7 EFLOP. Stronger teachers just shifted the OPD curve rightward—more expensive, no higher ceiling. There is also a hidden cost: students distilled from external teachers inflate response length, paying more FLOPs per step.
The right teacher: where the 50x comes from
The twist is the teacher's provenance. Among all teachers tested, the only setup where the student fully recovers teacher performance is using the student's own GRPO checkpoint as teacher. On math, OPD matched that checkpoint's accuracy with 50x fewer distillation-stage FLOPs (66x at the default 8-rollout setting), and 135.6x fewer than using an external Qwen3-8B teacher. The result replicates across student sizes, on code, and in a second model family (OLMo-3-7B)—where the student even beat its teacher by 11 absolute points.
But the paper's most honest accounting sits right here: on a single domain, the GRPO checkpoint already is the model you want, so distilling it back into the base student is pure added cost—the 50x only covers the distillation stage. Swapping teachers is not a free lunch; it changes what the ledger counts.
The real use case: specialize, then consolidate
Multi-domain is where distillation earns its keep. Train separate GRPO experts for math and code—no single checkpoint serves both—then merge them into one student with multi-teacher OPD. The distillation stage costs only 5.2% (two domains) and 8.0% (four domains) of the specialists' training FLOPs. Counting both stages, the two-domain pipeline matches joint GRPO with 42.0% fewer total FLOPs (31.6% on four domains), and joint GRPO never reaches the merged student's performance within the tested budget. Meanwhile, a single external 8B teacher on the multi-domain mix collapsed outright: the student degenerated into unstoppable repetition and emitted Chinese characters for English prompts.
The weight of this result: DeepSeek-V4 and Kimi K3 reports already bake multi-teacher OPD into their post-training pipelines as an engineering choice, with no full-pipeline accounting. This paper supplies the receipt—distillation's role is not "a faster upgrade than RL" but "a cheap consolidator that welds multiple experts into one model."
So what
Next time you hear "distillation is more efficient than RL," ask two questions: whose checkpoint is the teacher, and which stage's FLOPs are being saved? The likely answer: RL-train each domain into a specialist first, then merge with distillation an order of magnitude cheaper. Compute allocation shifts from "distillation or RL" to "both, each doing its own job."