Multi-token prediction (MTP) has become standard equipment in frontier pre-training — the paper names DeepSeek-V3/4, NVIDIA's Nemotron 3, GLM-5 and Kimi K3 as adopters. The extra heads improve downstream performance and double as speculative-decoding drafters at inference time. But a single-author paper from MIT, posted on arXiv on October 28, argues that standard MTP training buries a low signal-to-noise residual in its gradient — one that interferes more as models grow.
The residual: 80% of the gradient, lowest quality
The paper decomposes the MTP cross-entropy gradient exactly: a KL self-distillation term that pulls each MTP head toward the model's own next-token distribution (precisely what speculative decoding wants), plus a residual — effectively the next-token cross-entropy gradient against the data — with low SNR that buries the useful signal. Measured across training, the residual's SNR runs about 4x below the KL component, and the gap widens from 4.1x at 167M to 4.6x at 1.2B. Yet that residual carries roughly 80% of the MTP gradient reaching the shared trunk. A unit-step probe makes the split visible: stepping along the KL component lowers the next-token loss; stepping along the residual raises it — it is not just noise, it works against the main task.
The fix: one line of code
The obvious alternative — minimizing an explicit KL — would mean materializing vocabulary-sized logit tensors, which is memory-hungry and incompatible with the fused cross-entropy kernels modern pre-training relies on. The author's KL gradient-recovery trick sidesteps it: when two distributions share a linear output head, the KL gradient can be recovered exactly by combining the two cross-entropy gradients through straight-through gradient routing. Soft-MTP is a one-line change: no new parameters, no extra logits, fused-kernel compatible. On an H200, throughput lands within 0.07% of standard MTP, with identical FLOPs and peak memory; the repo ships a float64 test verifying machine-precision equivalence to explicitly computing the KL. Both DeepSeek-style and Meta-style MTP are supported.
The ledger: wins at every scale, fine-tuning most of all
Across four sizes — 167M, 332M, 689M and 1.2B (each trained on 31.5B FineWeb-Edu tokens) — Soft-MTP beats standard MTP in the paper's runs at every scale: zero-shot average gains widen with scale — about 0.1 points at small sizes, about 0.5 points at 689M and 1.2B, overtaking the NTP baseline at 1.2B (57.31, vs 56.80 for MTP and 56.75 for NTP); speculative-decoding acceptance length improves 1.8–2.2% averaged over five domains, speedup by 1.4–1.5%. The headline is fine-tuning transfer: after fine-tuning the 1.2B model on 21B tokens of Stack-Edu Python or 4.2B tokens of OpenMathInstruct-2, Soft-MTP takes seven of eight cells, leads HumanEval pass@64 by 3.1 points (43.90 vs 40.85), and is the only objective never below the NTP baseline. Against the explicit KL-MTP variant it matches or exceeds most metrics, while KL-MTP costs 1.5% more FLOPs, 5.6% lower throughput and 21 GB more memory.
The caveats: single author, rented GPUs, frontier scale untested
This is a single-author preprint; the experiments were mostly self-funded on 8 RTX PRO 6000 GPUs rented from Vast.ai, the validation ceiling is 1.2B parameters and 31.5B tokens, and "free lunch" is the author's own phrasing. KL-MTP still leads math fine-tuning (GSM8K 50.34 vs 49.66). Section 7 adds context: DeepSeek-V4.1-Flash has already dropped MTP for a post-trained DSpark drafter, so MTP's own future is open. The recovery trick, though, applies wherever a teacher and a student share a linear output head — including post-training self-distillation such as OPSD.
For teams still using MTP, this is a near-zero-cost engineering upgrade; for everyone else, the lesson is that even "standard equipment" can hide gradient-level flaws nobody had accounted for. Code and checkpoints for all four scales are open-sourced on GitHub.