For two years the speculative decoding literature has been chasing better draft models, deeper trees and wider trees. Almost nobody has touched the quantity that actually caps every method: how many target tokens one draft decision can bank. This paper calls that quantity the "unit", and shows it is nailed to 1 by a structural property of BPE. In Qwen3-4B's vocabulary "United States of America" is four tokens and no adjacent pair of them exists as one entry, so a multi-word phrase is out of reach of any method that drafts in the target's own vocabulary. The paper is authored by Madhusudhana Naidu and was submitted to arXiv on November 9.
A constant nobody questioned
The diagnosis starts from a formula: the tokens a forward pass can bank equal 1 + landing x (target tokens banked per accepted draft unit). That leading 1 is the target model's own bonus token; what gets multiplied is the unit. Every mainstream method — Medusa, EAGLE-3, DFlash and DSpark — drafts in the target's own vocabulary, so one node of the candidate tree is one target token. The unit is pinned at 1 and the only lever left is the number of nodes.
The authors hit this wall themselves first. Their earlier word-level drafter on Qwen3-4B reached a landing of 0.711, which does not look bad, but still banked only 1.049 target tokens per accepted guess — right against the floor of 1. The paper's summary is blunt: acceptance was not the limit; the unit was.
Why is the unit short? BPE's pretokenizer splits on whitespace before merges are learned, so no amount of BPE training can produce a multi-word token. SuperBPE makes the same argument from the tokenizer side: whitespace is not a reliable delimiter of meaning, and the restriction is an inherited convention rather than a design requirement. It solves the same problem by changing the target's own tokenizer, at the cost of pretraining from scratch — 8B models and roughly 330B tokens — whereas this work never touches the target. EAGLE-3's ceiling is set by the same unit: its draft head works over a truncated vocabulary of 32,000 ids out of 151,936 on Qwen3-4B, or 21.06%, and a target token outside that set is undraftable at any tree size or depth.
Changing the unit: a separate phrase vocabulary
The mechanism: the target model and its tokenizer stay frozen, while a two-LayerNorm MLP head reads the target's hidden state and predicts a one- to three-word phrase as a single entry in a separate draft vocabulary. A greedy longest-match expansion maps that entry back into the target's own tokens, and multiple candidates are verified in one forward pass behind a prefix-tree mask. Because a candidate is accepted only while it matches the target's own argmax, the served output is identical to greedy decoding.
The parameter arithmetic is clean: the head's fixed cost is 1,377,792 parameters (at d_model = 2560, H_DIM = 256), plus one output row of 257 parameters per vocabulary entry. So the draft vocabulary size is the parameter budget, exactly. 8,000 entries come to 3,433,792 parameters, 0.0854% of Qwen3-4B, with the output layer accounting for 59.9% of that; 80,000 entries come to 21,937,792 parameters, 0.5454%, with the output layer at 93.7%. A tenth of the vocabulary, in other words, buys a sixth of the parameters.
On results: the 21.94M head — 0.545% of Qwen3-4B — reaches 2.391x on GSM8K, 2.444x on MATH-500 and 1.544x on MT-Bench, where that head saw only maths distillation during training, so MT-Bench is zero-shot transfer. Shrunk to 8,000 entries, the head is 3.434M parameters and 0.0854%, 64x smaller than EAGLE-3's 218.2M drafter, and it retains 92.3% of the larger head's GSM8K tokens per pass.
The head-to-head is the more interesting number. Against EAGLE-3 run on one shared baseline, the 22M head reaches 102.3% and 108.5% of EAGLE-3's tree-16 on GSM8K and MATH-500, but on every set it stays behind tree-60 (91.5% and 98.1%, and only 66.6% on MT-Bench). The paper does not hide this: the claim is parameter efficiency at a comparable operating point, not a new state of the art in raw speedup.
Where the speedup actually comes from
This is the paper's most valuable self-audit, and the most counterintuitive part.
The authors counted what is actually inside the draft vocabulary. In the 8,000-entry head, 3,997 entries are multi-word, and only 467 of them (11.7%) are pure letters and whitespace; 43.9% contain a LaTeX character, 55.0% a digit and 24.7% a newline. The most frequent entries are identical across the two heads, and they are markdown and LaTeX scaffolding.
Then they logged which vocabulary entry banked each accepted block. Punctuation accounts for 31.4% on GSM8K and 41.0% on MT-Bench; LaTeX for 26.0% and 11.7%; plain English words only 7.3% and 22.6%. On maths, punctuation and LaTeX together supply 57-63%, and 59% of everything banked comes from 3-word entries which, per the census above, are mostly LaTeX fragments. The paper's own words: the method is harvesting the low-entropy formatting skeleton of mathematical prose.
Out of domain the bill inverts. On MT-Bench, LaTeX falls from 41.3% of banked tokens to 11.7%, plain words rise from 5.5% to 22.6%, single-word entries from 12.4% to 35.7%, and landing drops from 0.817 to 0.509. The explanation is that there is no formatting skeleton in conversation to harvest, so the drafter falls back on short common words.
One more detail stings: the 8,000-entry head beats the 40,000-entry head on MT-Bench at every N, 1.653 against 1.588 at best. The extra 32,000 entries were not merely unused on chat but harmful, each one a competing logit trained on the same supervision.
The other end: where words get shredded
The paper argues the real terrain is not English but the languages whose words get shredded most. Measured over 23 languages on Qwen3-4B's own tokenizer: English runs at 1.38 tokens per word, Hindi at 5.00, Telugu at 11.73, for an Indic average of 8.68. And of Qwen3's 151,669 tokens, only 253 are Indic-script tokens (0.17%), so a word-denominated draft banks several target tokens per decision for free. But the paper admits this case is largely unrun: the Indic experiments used Gemma-3-1B, and Gemma-3 spends 3.05 tokens per Telugu word against Qwen3's 11.73, a 3.8x difference that lands the measurements on the low-fertility side, exactly where the headroom is smallest.
The paper's own accounting
The limitations section is more honest than most work in this area, and every item is measured rather than anticipated. The denominator is fragile: the library version alone is worth 5.7-9.1% of a speedup number, and across three runs the same arm on the same card spans 5.7% on MATH-500. One card, one seed, 50/50/40 prompts, no confidence intervals. Online adaptation is not yet a result: landing rises monotonically from 0.736 to 0.759 to 0.764, but speedup has not followed (2.180 to 2.214 to 2.201), and there is no frozen control, so a rising curve cannot be separated from drift in the stream.
So what
The real contribution here may not be the 2.444x number but the move that turns the "unit" from a constant into a variable. If acceptance rate is not the bottleneck, the ceiling depends on how many target tokens one draft decision can bank — and that number is pinned by the whitespace pretokenization rule in the tokenizer. It is also a reminder to ask one extra question of any such speedup figure: is it buying back semantics, or formatting?
The applicability is fairly specific. On traffic with highly regular formatting — maths, code — a small 3.4M-parameter head, 0.0854% of a 4B target, buys more than 2x throughput; on conversational traffic the paper's own accounting shows the gain dropping to around 1.5x. So what to measure is not tree width but the fertility of your own tokenizer in the target language.