Training a draft model for speculative decoding usually means cross-entropy or KL divergence. But what actually governs the speedup at inference time is the expected acceptance length (EAL) — how many draft tokens survive verification per round on average. A single-author paper from the University of Hong Kong, REAL, takes this misalignment apart: not only do conventional objectives optimize something else entirely, even recent block-level objectives designed specifically for EAL carry a systematic bias that can flip the gradient direction.
Multiply Before You Average
Block-level objectives such as WTV multiply per-step acceptance rates to approximate the survival probability over the window. The problem is that an accepted token has a double identity: it extends the accepted prefix, and it determines the prefix on which the next draft token gets verified. Averaging first and multiplying afterward discards this branch structure, while the exact objective weights each path before averaging — and the difference is exactly the covariance between path weights and downstream acceptance rates. The paper's counterexample is crisp: in a two-step setting with a two-token vocabulary, the first-step target distribution is 0.4 versus 0.6 and the draft distribution is 0.8 versus 0.2, with the second-step target depending on which branch survives. Enumerating all four paths directly, the exact objective's gradient is positive and rewards raising the draft parameter, while the surrogate's is negative and rewards lowering it; at θ=0.5 the two objectives take equal value with opposite gradients. Nor is this a toy concern — the author logs this covariance throughout training: it stays positive, meaning the surrogate systematically underestimates the true objective.
REAL: An Unbiased Estimator, Loss-Only Change
REAL samples trajectories from the frozen target, multiplies each token by a path weight to recover the propose-and-accept branch distribution, and obtains an unbiased estimator of EAL. Rao–Blackwellization then replaces the final sampled factor of each term with its conditional expectation, cutting per-term gradient variance by 76–84% at position 2 and aggregate gradient variance by 52–58%; an auxiliary cross-entropy term stabilizes training from scratch. Ablations show pure cross-entropy and the CE-free variant both fall short — each component contributes. Every change lives in the loss function: drafter architecture, training pipeline, and inference procedure all stay untouched, with no additional training overhead.
The 4.792 Scorecard and Its Edges
The experiments freeze Qwen3.6-35B-A3B (a MoE model with a hybrid GDN structure, chosen to mirror current mainstream architecture) as the target, pair it with a Domino drafter, run one anchor plus a γ=7 draft window per round, and train for 3 epochs totaling 140K steps on OpenPerfectBlend prompts regenerated by the target at temperature 1.0. Evaluation runs on the vLLM engine across 9 task subsets and 5 random seeds, under both probabilistic and greedy serving protocols. Under probabilistic serving, REAL reaches 4.792 acceptance length, with the variant dropping Rao–Blackwellization at 4.791 — ahead of static LK at 4.740, WTV at 4.715, VSD at 4.657, and DBLast at 4.749. The best score on every one of the nine subsets belongs to one of the two REAL variants. At position 1 everyone is nearly tied; the gap opens later — acceptance retention at position 7 is 0.315 versus 0.306 for static LK. The advantage also accumulates: the lead over WTV grows from 0.041 after the first epoch to 0.077 by the third, and continuing into a fourth epoch REAL reaches 4.816 with no sign of saturation.
Two boundaries must be stated. Under greedy serving, static LK actually tops the table at 4.544 against REAL's 4.537; and when the sampling temperature drops to 0.5, static LK overtakes REAL 5.002 to 4.993. REAL's edge lives in stochastic sampling — the closer decoding gets to deterministic, the narrower it becomes. That is the paper's own stated scope, not a defect being hidden. Validation also covers a single target, a single drafter, and a single data recipe; the VSD and DBLast comparisons are loss-level adaptations rather than full system reproductions.
So What
The real contribution here is not a few hundredths of acceptance length but a methodological warning: whenever a training metric stands in for the true objective, ask first whether the proxy is unbiased. The author measured the standard training-time EAL proxy favoring WTV throughout training while the true serving ranking says the opposite — unsurprising, since that proxy shares the same product-of-marginals form as WTV; the metric was quietly lying. In engineering terms this is a drop-in loss, with a reference implementation on a fork of the vLLM speculators library (King-ty/speculators). Concurrent efforts such as EDR and BV loss attack the same problem from dynamic-programming and block-verification angles — the direction is heating up. For teams treating speculative decoding as their main inference cost lever, a correction at the objective level deserves attention before yet another layer of systems tricks. That said, the validation covers a single target and a single drafter (paper) — extrapolate with care.