Why a Stronger Teacher Can Fail: Lightning OPD 2.0 Removes Style Noise from Distillation

Large-model distillation has an uncomfortable paradox: replacing a teacher with a more capable model does not necessarily make the student better. In some cases, the student barely improves at all. A new NVIDIA paper, Lightning OPD 2.0, argues that the problem is not simply teacher quality. It is the hidden mismatch between how the teacher expresses an answer and how the student reference model was originally trained to express one.

How style contaminates on-policy distillation

On-policy distillation, or OPD, asks a student model to generate reasoning trajectories and then lets a stronger teacher score the generated tokens. Unlike reinforcement learning methods that may provide only a sparse sequence-level reward, OPD supplies dense token-level feedback. That makes it an attractive and comparatively stable way to improve reasoning models.

Lightning OPD previously reduced the systems cost of this process by precomputing student rollouts and teacher log probabilities, then reusing that frozen cache during optimization. The teacher no longer has to remain online throughout training.

However, effective OPD has depended on what the authors call teacher consistency. Ideally, the model that supplies OPD supervision should be the same model that generated the demonstrations used to build the supervised fine-tuning, or SFT, reference policy. In production pipelines, this assumption is often unrealistic. SFT datasets may combine demonstrations from several models, their provenance may be incomplete, or a team may intentionally use one model to generate demonstrations and another, stronger or cheaper model for later distillation.

The mismatch matters because a token-level probability difference does not cleanly mean that a reasoning step is wrong. It can also mean that the new teacher prefers different wording, formatting, transitions, derivation length, or reasoning cadence. A teacher may assign low probability to a valid phrase such as “therefore” simply because it would have written the transition differently. Standard OPD treats that disagreement like corrective evidence. Since stylistic preferences recur across thousands of trajectories, they can accumulate and drown out the context-specific signal that actually identifies reasoning errors.

This explains why a nominally stronger teacher can provide surprisingly weak supervision: part of its gradient is spent teaching its voice rather than its reasoning.

The Lightning OPD 2.0 correction

Lightning OPD 2.0 does not attempt to label style tokens manually. Instead, it exploits a statistical distinction: style preferences tend to repeat across unrelated prompts, while useful reasoning corrections depend more heavily on the current problem and reasoning state.

The method begins with the cached log-probability difference between the selected OPD teacher and the SFT reference for every realized token. It then splits cached rollouts into prompt-level folds. For each held-out fold, the other folds are used to build two smoothed lookup tables.

The first table estimates recurring disagreement by token identity, capturing lexical preferences. The second estimates disagreement using coarse context coordinates: normalized position within the response and surprisal under the reference policy. These coordinates distinguish, for example, a common token near the start of an answer from an unusual token late in a derivation without conditioning on the full prompt.

The two estimates are averaged to form an operational proxy for recurring style bias. That estimate is subtracted from the raw teacher-reference disagreement before constructing the OPD update. Because the current rollout is excluded from its own lookup tables, cross-fitting reduces self-fitting: the method cannot explain away a token using evidence from the same response.

The authors are careful about the interpretation. The removed component is not a perfect semantic classification of style, and every residual is not guaranteed to be correct reasoning evidence. It is a practical estimate of the part of teacher disagreement that is predictable across other rollouts.

What the benchmarks show

The experiments use Qwen3-30B-A3B-Thinking-2507 as the independently selected OPD teacher in two settings. The first starts from Qwen3-4B-SFT, whose demonstrations were generated by Qwen3-8B. The second starts from Klear-Reasoner-8B-SFT, trained with long-chain-of-thought data distilled from DeepSeek-R1-0528.

On the 4B reference, Lightning OPD 2.0 raises average mathematical reasoning performance from 48.3% to 51.7% and average code-generation performance from 32.6% to 35.7%. The original Lightning OPD reaches 48.6% and 34.3%, respectively, showing that most of the additional gain comes from correcting the cross-teacher signal rather than merely applying more distillation.

On Klear-Reasoner-8B-SFT, the method raises the average mathematics score from 73.6% to 74.6% and the average code score from 54.9% to 58.5%. Individual results include 82.4% on AIME 2024 and 63.0% on LiveCodeBench v5.

The mechanism analysis is more important than any single leaderboard number. Using an absolute deviation threshold of one nat, residualization reduces the share of high-deviation tokens from 8.14% to 3.85% in the Qwen3-4B setting. In the Klear-Reasoner setting, it falls from 7.19% to 2.02%. The corrected cross-teacher signal therefore moves substantially closer to the teacher-consistent diagnostic signal.

Ablations also support the design. Token-only and context-only estimates both help, while the full combination with prompt-level cross-fitting produces the strongest overall point estimates. The result suggests that lexical repetition and coarse response context capture different parts of the mismatch.

Why this matters beyond one benchmark

The real contribution is architectural at the training-pipeline level. Lightning OPD 2.0 begins to decouple two decisions that had been artificially tied together: which model should generate SFT demonstrations, and which model should teach during later OPD. A team can preserve an expensive existing SFT dataset while independently selecting a more suitable distillation teacher, rather than regenerating demonstrations and rerunning SFT every time the teacher changes.

That flexibility matters as organizations mix proprietary and open-weight models, reuse datasets with imperfect provenance, and optimize separately for quality, cost, latency, and licensing. It also reframes teacher selection. The best teacher is not simply the model with the highest benchmark score; it is the model whose useful capability signal can reach the student without being overwhelmed by incompatible expression habits.

There are clear limitations. The paper evaluates two Qwen-family student settings, one selected teacher, and mathematics and code-generation tasks. The authors also note that code will be released soon, so independent replication across tokenizers, model families, and more open-ended tasks remains necessary.

Still, the central lesson is durable. Post-training quality depends not only on how much the teacher knows, but also on whether the learning channel separates knowledge from presentation. As models become better at answering, the next competition is making them better at teaching.