The inference bottleneck of video diffusion transformers is no secret: 3D full attention scales quadratically with sequence length, and at 720p with 81 frames every transformer layer and every denoising step pays that bill again. AB-Attention (arXiv:2610.39663, posted 2026-10-29) comes from the InfRec team at CUHK-Shenzhen together with Shandong University, the University of Tokyo, Fudan, UC Berkeley, HKUST and NUS. The idea: no training, no weight changes — just skip 89–92% of attention computation. The paper reports up to 2.23x end-to-end speedup while improving PSNR by up to 4.4 dB over the strongest sparse baseline.

Three moves: score, sample, pack

The method has three components. The Intra-Block Clustering Selector (IBCS) clusters queries only within each query block and uses the weighted centroids to score every key/value block — no token permutation, no global clustering, so scoring stays cheap. Sampling with Inclusion-Probability Weighting (SIP) is the core: instead of hard truncation, key/value blocks are sampled with probability proportional to their scores and reweighted by inverse inclusion probability. The paper shows this single sparse pass yields unbiased estimates of both the attention numerator and the softmax normalizer, so dropped blocks never vanish from the normalization. Finally, a paged kernel gathers the scattered selected blocks into full tensor-core tiles, refining selection granularity at no extra tensor-core cost.

The paper's critique of existing dynamic sparse methods lands on three points: block-importance estimates are either coarse or costly; pruned blocks drop to zero in the softmax; and keep-or-drop decisions happen at whole-tile granularity. In plain terms, the root of distortion is often not "selecting the wrong blocks" but the act of truncation itself.

Main results: sparser, yet more faithful

The evaluation covers three text-to-video diffusion transformers — Wan2.1-T2V-1.3B, Wan2.1-T2V-14B and HunyuanVideo-T2V-13B — at 720p, with 81 frames on Wan2.1 and 129 on HunyuanVideo, over 600 prompts from the Penguin Benchmark, each sparse run compared against a dense reference with the same prompt and seed, on NVIDIA H20 GPUs in BF16 with PyTorch 2.6. Key readings:

  • Wan2.1 1.3B: 89.16% skipped, per-video latency down from 962.39s to 479.11s (2.01x), PSNR 34.436;
  • Wan2.1 14B: 89.16% skipped, 4743.52s to 2571.52s (1.84x), PSNR 30.788;
  • HunyuanVideo 13B: 92.35% skipped, 4399.03s to 1974.61s (2.23x), PSNR 34.347.

On the baseline side, the static-mask RadialAttention reaches 73.03–73.63% sparsity and the dynamic SVG2 and DFSAttn sit between 67.89% and 81.29%; the authors execute all baselines' masks on FlashInfer's FlashAttention-3 block-sparse backend to rule out implementation differences. The baselines still compute 1.8–3.6x as much attention yet trail by 1.6–4.4 dB in PSNR. Across the five VBench dimensions (aesthetic quality, background consistency, imaging quality and more), AB-Attention stays close to the dense reference.

Ablation: inverse-probability weighting carries 4 dB

At 14B, 100 prompts and the same 89.16% sparsity, the paper swaps out one component at a time and produces the most informative numbers in the text: removing SIP's inverse-probability weighting costs 4.00 dB PSNR; switching to uniform sampling without weights costs 7.25 dB; switching to Top-k hard selection costs 3.99 dB. Meanwhile, replacing intra-block clustering with plain query mean-pooling costs only 0.91 dB, and tile-level selection costs 1.02 dB. The bulk of the error is therefore not in "which blocks you pick" but in "how you treat the ones you don't" — the selector can afford to be plain; truncation must be replaced by probability-corrected sampling estimation.

Caveats: H20-only kernel, code not out yet

The Limitations section is candid: the kernel is built for the H20 GPUs used in the experiments, supports forward inference only, and the evaluation covers just three text-to-video models. The GitHub repository currently holds a README and a license with a "Code coming soon" badge — third-party reproduction has no path yet, and every number above is author-reported. The timed region covers text encoding, all 50 denoising steps and VAE decoding, excluding model loading; even after the 2.23x speedup, a single video still takes 8–43 minutes, far from real time.

So what: the frontier of sparse attention is shifting from "select more accurately" to "estimate more honestly". AB-Attention turns "skipping" into "unbiased estimation" via importance sampling, and the idea is not specific to video diffusion — long-context LLM prefill and any softmax-normalized sparsification could, in principle, adopt it. Once the code lands, the immediate question is how it really performs on hardware beyond the H20.