arXiv 2606.18394 introduces JetSpec, a speculative-decoding method that breaks the long-standing 5-6× speedup ceiling. The core innovation: a parallel-tree draft model that generates multiple candidate token trees in a single forward pass, paired with a tree-structured verifier that validates the candidates in parallel. On H100, JetSpec hits 9.64× end-to-end speedup on Llama-3-70B — a record for speculative decoding.
The traditional bottleneck: speculative decoding's speedup is limited by the "drafter overhead" — the time to generate candidate tokens. Even with a small drafter, this overhead is not negligible, and the 5-6× ceiling is a "drafter-speed" ceiling, not a "verifier-speed" ceiling. JetSpec's fix: the drafter is no longer a sequential model but a parallel-tree generator — it produces a tree of candidate tokens in a single forward pass, with the tree depth and width dynamically controlled by a confidence estimator.
The verifier also gets a parallel upgrade: a tree-structured attention mask allows the verifier to validate all candidates in parallel, with the rejection sampling done in a single GPU kernel.
Experimental results: on H100 with Llama-3-70B, JetSpec hits 9.64× end-to-end speedup; on A100 it's 7.2×; on consumer 4090 it's 5.1×. The speedup is particularly notable on long-output tasks — code generation, long-form Q&A, and chain-of-thought reasoning.
The bigger signal: speculative decoding has entered the "parallel-tree" era. The traditional "small drafter + serial verifier" paradigm is being replaced, and the next round of competition will be in the "tree quality" and "verifier efficiency" fronts. For the industry, this means LLM serving costs can be cut by another order of magnitude, and "inference economics" will become a key moat for model API providers.