The story of long-context LLMs over the past two years has essentially been a fight against two walls: the memory wall, kept at bay by KV-cache compression and FlashAttention-style IO optimization; and the compute wall, tamed by chunked prefill, speculative decoding, and MoE. But at 256K context length, the two-stage flow in FlashAttention — "scan once to find the top-k critical blocks, then compute those blocks" — exposes an awkward truth: deciding which blocks to compute already requires scanning every query-key pair, so the memory overhead is essentially the same as full uncompressed attention. A paper titled "Faster Than Flash" (arXiv:2609.00097) from Qiu Xipeng's group at Fudan, accepted at ICML 2026, pushes that seemingly immovable ceiling a notch higher.
The Hard Constraint on an Old Problem
Sparse attention is the mainstream path for long-context decoding: if only a few blocks of the query-key matrix actually matter, why compute all of them? The catch is figuring out which blocks matter. The naive approach uses each query's historical max attention score per block as a "scoring tag" — but that tag requires externalized metadata and GPU-wide synchronization. On bandwidth-starved cards like the H100, the read/write traffic for the tag alone eats most of the sparse-compute savings. Faster Flash Decoding (FFD) takes a first cut: it stuffs the selector and the computer into a single fully fused kernel, with no metadata sync in between. How does the selector work? A low-bit-quantized query scans blocks on the fly, and the scan output is fed straight into the attention compute — no round trip to global memory, and no global sort for top-k truncation.
top-delta: Tearing Down "Global Synchronization" Too
FFD's second key design is the top-delta strategy. When queries are distributed very unevenly across long context segments (think: caring only about the last 4K of a 256K prompt), traditional approaches either "select everything" and waste compute, or "threshold everything" and miss long-range dependencies. top-delta instead keeps the top-d blocks by score plus every block whose score gap from the top is within δ. Each query decides d locally, no GPU-wide barrier required. On RULER and LongBench, key retrieval-style tasks barely lose accuracy. The cost — scan work growing linearly with context length — is absorbed by bandwidth savings once everything lives in a fused kernel.
The Promise and the Boundaries in the Numbers
Per the paper, FFD reaches up to 11.6x kernel-level speedup and 2.37x end-to-end throughput, with context pushed cleanly to 256K. The biggest deal is the "training-free + plug-and-play" property: no retraining, no weight changes, drop it onto any already-trained LLM. Code is open-sourced at github.com/qluoluo/faster-flash-decoding. But the boundary lines are worth flagging: the 11.6x kernel speedup is a peak, not an average. Accuracy is validated only on RULER and LongBench — whether real workloads (codebase understanding, long-video captioning, agentic cross-page operations) still hold up is something we will only know after running them. FFD trades memory for bandwidth; stack it with KV-cache compression and stability on cards older than the H100 is also something to measure before deployment.
So What
The next leg of long-context progress will likely live on the "Faster Than X" line. FlashAttention pushed IO optimization to 3.0; what comes next is selector-computer fusion, scan-compute reuse, and distribution-adaptive sparsity. FFD is not "yet another 2x" — it deletes work that used to be mandatory. That kind of architectural subtraction is worth watching more closely than yet another pure-kernel optimization.