Reinforcement learning has trained the industry to look for a stronger teacher: find one, then have it supervise a reasoning model. But the standard self-distillation recipes all share an awkward precondition — the teacher has to be built out of privileged information. Reference solutions, expert demonstrations, environment feedback. When all you have is unlabeled prompts, that path closes.
A paper posted to arXiv on October 28 inverts the construction. Instead of strengthening a model into a teacher, it weakens the model on purpose and treats the gap between the strong copy and the weak copy as the training signal. The method is called WISD (Weakening-Induced On-Policy Self-Distillation), authored by Yecheng Wu, Song Han and Han Cai of MIT and NVIDIA, with code released alongside (arXiv:2610.38681, project page, GitHub).
Flipping the pair: a free teacher signal
The motivation is straightforward. Existing self-distillation methods such as OPSD and SDPO feed the teacher reference solutions, expert trajectories, or environment feedback, so training cannot proceed without privileged data. WISD swaps the two ends of the pair: the original model π₀ becomes the stronger member, while the weaker member π_w is derived from the original model's own weights, either through low-bit quantization or through magnitude pruning.
Neither construction needs calibration data — both operate directly on weights, in groups of 128 input columns (default b=3 with RTN quantization) or by threshold (default τ=1/6 for pruning), while embeddings, attention heads, norms and routers stay untouched. The supervision signal is then simply the token-level log-probability contrast r_w(a|s) = log π₀(a|s) − log π_w(a|s): tokens the original model prefers over its weakened copy are treated as the direction to move.
The numbers across five models
Evaluation covers five reasoning models, from 4B to 35B and spanning dense plus mixture-of-experts designs: Qwen3-4B, Qwen3.5-4B, OLMo-3-7B-Think, Qwen3.6-27B and Qwen3.6-35B-A3B. The benchmarks are three math sets (AIME 2024/2025, HMMT 2025, Avg@32) and two code sets (LiveCodeBench v5 and v6, Avg@4).
The headline results: the quantization variant lifts average accuracy by 4.0 percentage points and the pruning variant by 5.5 points, with positive gains in 21 and 23 of the 25 model-benchmark settings respectively.
The largest gains land on Qwen3.5-4B, where quantization adds 11.3 points and pruning 13.4 points to the five-benchmark average. HMMT 2025 goes from 57.7% to 70.5%, and LiveCodeBench v6 from 39.1% to 56.7%. The effect survives at larger scale: pruning lifts Qwen3.6-27B's five-benchmark average by 5.2 points, and quantization lifts Qwen3.6-35B-A3B by 5.9 points (on that model, HMMT 2025 rises from 68.5% to 74.6% and LCB v6 from 61.8% to 69.5%).
The comparison group is the more revealing part. On Qwen3-4B, the untouched model's five-benchmark average is 55.98, while OPSD — which does use privileged information — reaches only 52.70, and SDPO only 49.88. Both land below the untrained baseline.
Ablation: weaken just the right amount
The paper surfaces a pattern worth noting: the relationship between weakening strength and downstream gain is not monotonic. Going from 8 bits to 3 bits of quantization moves Qwen3.5-4B on HMMT 2025 from 56.04% up to 68.33%, and dropping further to 2 bits collapses it back to 49.90%. Pruning behaves the same way: 60.94% at τ=1/16, a peak of 70.52% at the default τ=1/6, then 49.58% at τ=1/2.
The authors' reading is that weakening too gently yields too little contrast, while weakening too aggressively destroys the weak model's own reasoning ability until noise swamps the signal. At the best settings, the distilled student also produces shorter answers than the baseline.
Is "weaken it somehow" enough? Table 3 says no. Simply rewarding the model's own confidence (Self-enhancing), or using a temperature-softened copy as the weak teacher (T=2), stays within 0.6 points of the original. What works is structured degradation at the weight level.
Where the signal lands
Using the ThinkARM taxonomy to label 64 Qwen3-4B math trajectories, the paper measures how training signal is allocated by the share of initial logit-gradient L1 magnitude. Exploration holds 14.8% of tokens but absorbs 22% of the signal; verification holds 8.1% of tokens and takes 11%. Routine implementation gets less. The contrast produced by deliberate weakening lands squarely on intermediate computations and logical checks.
The cold water
First, the arXiv record is marked "Work in progress." The code repository currently shows 1 commit, and the repository star count on the HF papers page reads 0, so third-party reproducibility is still a ways off.
Second, read the cross-method comparison carefully. The grey rows for OPSD, SDPO, PW-OPSD and LOPD in Table 2 are published results from their own source papers under their own evaluation setups — the project page explicitly says "treat them as a reference." They are not re-runs under one codebase and one dataset. So the paper's claim of state-of-the-art performance without privileged information is, strictly speaking, a comparison against published numbers rather than a unified re-run.
Third, the ablation absolutes sit on the single HMMT 2025 benchmark. The "too little, too much" shape is a judgment relative to the same baseline, not an end-to-end measure of scale benefit, and the paper offers no positive evidence on whether the method decays as models grow larger.
The method itself is still worth a look: each domain needs just 3,200 unlabeled prompts, four responses each, 12,800 cached trajectories and 200 optimizer updates, with the frozen strong-weak pair scoring the cache once. No scoring model stays live during training, and no annotation is required at all.
So what: the real contribution here is not a score but a different question. When no good teacher is available, manufacturing a bad one can become a zero-cost signal source. Quantization and pruning — techniques originally invented to save memory — get repurposed here as a free teacher-signal generator.