Autoregressive LLM generation speed limited? Draft-model token guessing is still a sequential-compute bottleneck. UCSD and Google jointly published DFlash on arXiv: bringing the block-generation idea of diffusion models into speculative decoding, achieving an average 3.13× speedup on TPU v5p, up to 6× on math and code tasks, already integrated into the vLLM TPU ecosystem.
The soft spot of speculative decoding: traditional speculative decoding uses a small model to guess and a large model to verify, but the draft phase itself still requires sequential token generation — guessing K tokens means running K autoregressive steps. The larger the model, the longer the sequence, the more obvious this nested bottleneck becomes. DFlash's core is migrating the "generate a whole block at once" idea from diffusion models into the token domain: the draft phase generates a whole block of draft tokens in parallel, the verification phase handles them in block units. Engineering-wise it relies on dual-cache architecture, power-of-2 padding to optimize CPU-TPU data transfer, and state synchronization to prevent sequence-length inflation. TPU v5p's K-Flat finding: verification cost is nearly constant for block sizes between 16 and 1024, making improving draft quality more cost-effective than increasing block size. Compared to EAGLE-3, DFlash achieves 2.29× end-to-end speedup, proving non-autoregressive draft generation is fully viable in LLM inference, with the potential to become a standard optimization for large-model deployment.