Diffusion language models (DLMs) are a serious candidate for the next text-generation paradigm: instead of decoding left to right, they denoise many token positions in parallel. But open the ledger and inference is not cheap. In block diffusion models (BDLMs), each generated block is refined over multiple denoising steps that repeatedly attend over an ever-growing historical prefix, and every position in the current block is pushed through the full Transformer stack at every step — even though only a small subset of positions is actually resolved at that step. A new arXiv paper, DUET (2611.09239), cuts both sources of waste with a training-free, dual-sparse scheme: at a 32K context length, SDAR-8B's per-step latency drops from 99.53 ms to 21.09 ms, a 2.41x end-to-end speedup.
Where the wasted compute lives
The paper splits BDLM redundancy into two parts. On the history side, every denoising step attends over the full prefix, so attention gets more expensive as context grows. On the block side, all positions in the current block traverse every layer at every step, though only a subset is finally selected for the current update — the rest burn deep computation (attention projections, FFN/MoE) for nothing.
Before building anything, the authors establish two empirical observations showing both costs can be cut:
- Historical attention is concentrated and cross-step stable. A prefix KV subset with budget 256 captures about 85% of the historical attention mass; raising the budget to 512 gets nearly 90%. Across the whole block's denoising trajectory, coverage varies by only about 2.5 percentage points. Select a compact subset once per block and reuse it — no per-step reselection needed.
- Update positions are predictable at early layers. Applying the model's own decoding policy to hidden states from early layers: at layer 1, retaining 70% of masked positions already recovers about 80% of the final update positions; 90% retention reaches about 95% recall. The "who gets updated this step" signal is already strong early, so there is no need to send every row to the last layer.
A training-free dual-sparse framework
DUET's two moves leave the model weights untouched.
Prefix KV sparsity: after the first denoising step of each generation block, it scores historical keys against block queries and selects one compact, block-shared historical KV subset (budgets 256/512/1024), reused across all denoising steps in that block; the next block re-selects against the full cache.
Early query sparsity: at a chosen early selection layer, the model's original decoding rule predicts which masked positions will be updated this step, and only those active rows propagate through the remaining layers. Deep attention projections and FFN/MoE computation are paid only for active rows; bypassed rows stay masked for the next step. Configurations differ per model: LLaDA2.1-mini uses retention ratio 0.7 with selection layer 1; SDAR-8B uses 0.5 with layer 5.
No extra trained predictor is introduced — reusing the model's own decoding policy is what keeps the method training-free.
Results: fast steps are not fast generation
On quality, LongBench Macro-F1 at budget 512: LLaDA2.1-mini scores 51.35 versus 51.33 dense. Across four shorter benchmarks (budget 256), SDAR-8B matches or slightly beats dense on MMLU (74.53 vs 74.52) and HumanEval (62.20 vs 61.59), and LLaDA's HumanEval rises from 78.66 to 80.49. The cost is not zero: LLaDA's MATH-500 falls from 91.80 to 87.60.
Efficiency (80 NarrativeQA samples, 256 generated tokens, NVIDIA H800) tells the sharper story. At 32K, LLaDA2.1-mini gets 1.34x end-to-end and 2.83x per-step; SDAR-8B's gains grow with context — 1.53x, 2.05x, 2.41x at 8K, 16K, 32K — with end-to-end latency down from 18.416 s to 7.656 s and 4.72x per-step.
The baseline comparison is the instructive part. FOCUS also reaches a nice per-step latency (21.23 ms) but changes the decoding trajectory, pushing average steps from 25.05 to 64.53, so its end-to-end speedup shrinks to 1.04x. LoSA at 32K is end-to-end slower than dense (3.672 s vs 3.486 s), and its measured average KV usage of 434/405 exceeds its nominal budget of 256. The authors note all methods run in one inference framework with the same Triton kernels, keeping the comparison fair.
Engineering takeaways
The paper is a reminder for the inference-optimization crowd: a beautiful per-step latency means little if step count inflates — throughput must be measured end-to-end under real decoding dynamics, not just in kernel benchmarks. For the diffusion-LM line, DUET's contribution is decomposing "inference is expensive" into two verifiable structural facts (concentrated, stable attention; early-predictable updates) and harvesting them with minimal intervention. The paper releases anonymized source code and testing procedures; acknowledgments cite the Jiangsu Science and Technology Major Project (#BG2025005), and corresponding author Jingwei Xu has an nju.edu.cn address. Its AI-use statement is unusually candid: generative AI helped optimize implementation and evaluation code, monitor runs, and polish the manuscript, while the core method came from the authors — a disclosure habit worth copying.
For diffusion LMs to reach production serving, inference optimization has to evolve in lockstep with the architecture. DUET shows that sparsity touching no weights can already claim a large slice of the redundancy — the open question is whether these two observations generalize across the growing zoo of block-diffusion models.
Paper: DUET: Dual-Sparse Inference for Block Diffusion Language Models