Diffusion language models (dLMs) want to be the alternative to autoregressive generation, but their post-training has stayed crude: supervised fine-tuning treats every token of the final text as equally important teaching material, while online reinforcement learning spreads one trajectory-level reward across every position. Masked diffusion has a distinctive property here — a few early commitments during denoising sharply reduce the uncertainty over the remaining masked positions and effectively pin down the rest of the response. Failures are rarely wholesale; they trace back to a handful of critical decisions. Existing recipes ignore that signal entirely.

Train Only the Pivots

Pivot-SD is an offline self-distillation framework built around that signal. An information-gain metric measures how much each denoising commitment reduces uncertainty over the remaining masked block; the top-K highest-impact commitments become "pivots". Supervision is then applied to pivots only: pivots from successful trajectories are reinforced with cross-entropy, pivots from failed trajectories are suppressed with targeted unlikelihood, and the valid sub-steps inside failed trajectories are left untouched. That is a different granularity from SFT discarding failures wholesale or RL rewarding entire trajectories.

The 200-Question Ledger

The training budget is unusually small: 200 questions, four rollouts each, 1,000 steps. On LLaDA-8B-Instruct (256-token generation, mean of three runs, same budget as all baselines), Pivot-SD beats full-sequence SFT and budget-matched diffusion RL (diffu-GRPO, wd1++) on all four benchmarks:

  • MATH500: 31.40 base → 37.47, two points above SFT-GT's 35.47
  • GSM8K: 75.13 → 79.51
  • HumanEval+: 32.32 → 40.43, the largest gain of the four
  • MBPP+: 43.12 → 47.12

The compute ledger is just as direct: 800 rollouts sampled offline once for 2.5 hours plus 0.3 hours of training on 2× NVIDIA L40 — roughly one-eighth of the wall-clock of online RL, with average accuracy more than three points higher. The paper has been accepted as an EMNLP 2026 Main Conference Oral (arXiv:2610.03665).

Three Takeaways

  • Signal density beats data volume. 200 questions outperforming full-sequence SFT says a model's behavior is shaped by a few decisive steps, not token count. For domains with expensive verifiers and annotations, this is a budget-friendly route.
  • Failed trajectories finally get surgical use. SFT throws them away; Pivot-SD penalizes only the pivots that caused the error and preserves the valid sub-steps around them — negative samples get a precise price for the first time.
  • The limits are explicit. Experiments cover an 8B model, math and code tasks, and 256-token generations. Whether pivot selection stays efficient at longer horizons and larger scales is left open.

While the debate centers on whether diffusion LMs can catch up with autoregression, this paper points at a different lever: training signal should be allocated by information content, not spread evenly across tokens. The pivots are found — the next move is testing them on a bigger board.

Reference: paper arXiv:2610.03665 · project page sunwoohong.github.io/pivot-sd