Speculative decoding is one of the most practical LLM inference speedups today: a lightweight drafter proposes a block of tokens and the target model verifies them in one forward pass, cutting the serial waiting of autoregressive decoding; parallel drafters push the speedup further by drafting a whole block in one forward pass. A new arXiv paper, SpecPilot, argues that released speculative decoding systems deliver only part of the speedup they could — the remaining headroom is eaten by three engineering problems.
Three sources of wasted speedup
The paper names them. First, with a single request, each verification step accepts only a few tokens yet reads every target weight, so the weight-reading cost far exceeds what the accepted tokens earn. Second, once batching kicks in, per-request verification roughly halves the speedup over autoregressive (AR) decoding. Third, speculative decoding is deployment-specific: change the target model, drafter, GPU or serving regime and the optimal parameter mix changes — no universal tuning exists, which is exactly the part generic papers cannot deliver and engineering teams lack the manpower to do exhaustively.
An agent loop that tunes itself
The authors hand the tuning itself to an autonomous agentic loop. Each round, a cost model locates the largest gaps between the current system and its ceiling; parallel agents each explore one optimization direction and integrate the successful ones; a dedicated kernel sub-agent tunes the kernels furthest from their memory-bandwidth limit. Two safety designs stand out. Token forcing checks every output token against the stock system specifically to catch reward hacking, and a structured memory carries validated mechanisms to new rounds and new deployments instead of relearning them. Here stock means the released drafter system (DFlash2 or DSpark) without SpecPilot, served on the same engine.
Measured: more than double on the same hardware
The loop surfaced a set of general mechanisms: a lossless 12-bit weight format, tree verification for hybrid recurrent targets, two-level verification, and three batching mechanisms. The evaluation spans four target models (the public table includes Qwen3.8-27B and Muse-Glimmer-30B), three drafters (DFlash2 and DSpark among them), two GPU platforms (H100 and RTX PRO 6000) and five benchmarks (gsm8k, humaneval, math500, mbpp, mt-bench), with each request generating 2048 new tokens, per-request speedups geometrically averaged, and batch runs at concurrency 16, 20 and 24. Without training or quality loss — decoding stays greedy and the target model decides every token, strictly lossless — the speedup over AR rises from stock's 4.46x to 10.87x with one request, and from 1.94x to 3.93x under batching, more than doubling in every deployment. Tokens committed per verify step in single-request settings climb from roughly 4.4-5.2 to 12.6-13.1.
The code is open-sourced under Apache 2.0 (derived from SGLang), shipping the engine found for each of six deployments plus scripts to serve, benchmark and verify them. The paper lists ten authors, led by Kunjun Li: arXiv paper, GitHub repo.
So what
The takeaway worth keeping is not the 10.87x number but the engineering pattern it validates: deployment-specific systems optimization is too costly for humans to complete, and an agent loop can mine that headroom automatically. Just as telling is how the authors treat their own agent — however good the output, every token is checked against the stock system to catch cheating. For teams sensitive to inference cost, before waiting for the next model generation, it may be cheaper to measure how many multiples are still buried in the current deployment.