The sequential bottleneck of autoregressive decoding is being broken. On May 1, a paper published on OpenReview proposed the "Speculative Speculative Decoding (SSD)" algorithm, further parallelizing the prediction flow on top of speculative decoding, achieving up to 5× speedup.
Traditional speculative decoding uses a fast draft model to predict subsequent tokens, then has the main model verify them in parallel. But there's still sequential dependency between prediction and verification — the draft model must wait for current verification to complete before starting the next prediction.
SSD's core idea is to let the draft model "guess" the verification result while verification is happening, and pre-prepare candidates. When the actual verification matches the prediction, results can be returned immediately, eliminating drafting latency. The researchers implemented the algorithm as Saguaro, testing on open-source inference engines, achieving 2× speedup over optimized speculative decoding baselines, 5× speedup over pure autoregressive decoding.
The value of this work is that it's not another flashy engineering trick, but a rethinking of the bottleneck structure of speculative decoding at the algorithmic level. It points out the real problem isn't "can the draft model predict," but "what is the draft model doing while waiting for verification" — the answer is nothing, so let it do more at the same time.
Of course, the 5× speedup number comes from specific scenarios on open-source engines; actual deployment benefits will vary based on model size, hardware configuration, and request type. But the direction is clear: LLM inference's acceleration space hasn't been exhausted yet, and the next breakthrough may not be in the model itself, but in the redesign of the scheduling layer.