In a million-token conversation, the user asks about one detail from chapter three — yet global attention layers must scan the entire KV cache to generate every single token of the reply. The bottleneck of long-context inference is not parameter count but this per-step full-context memory traffic. A paper submitted on Sep 2 by KAIST AI and Google DeepMind (arXiv:2609.02737) proposes Declarative Attention (DA), with a counterintuitive idea: instead of using external scorers to predict which tokens matter, let the model itself declare, inside its own chain-of-thought, which region it needs to read.
The protocol: three attention modes, parsed like tool calls
DA partitions generation into three modes: <global> reads the full context for navigation, <focus> reads only the named segments, and <local> reads no context at all, doing self-contained reasoning over the response so far. The long input is split into "magic chunks" of roughly 2048 tokens, and the inference engine runs a state machine that parses these declarations — exactly like parsing tool calls — then rewrites the KV-cache block table in vLLM so the attention kernel simply reads less. No kernel modifications, no scheduler changes: the integration is a hook on the attention metadata builder. The mask is aligned to blocks (16-32 tokens) and applies only to global attention layers; sliding-window and linear-attention layers do not grow with context, so there is nothing to save there.
The key property is reversibility: DA masks but never evicts. The cache stays fully resident, and a <focus> step can be followed by a <global> step that re-attends everything — the structural reason accuracy holds up.
The numbers: 52% fewer reads, about one point of accuracy
Across 15 long-context tasks (RULER, LongBench v1/v2, LooGLE, ZeroScrolls) and six off-the-shelf models (Gemma-4-31B/12B/E4B, Qwen-3.6-27B, Qwen-3.5-9B/4B), zero-shot evaluation shows: on Gemma-4-31B, total attended tokens during decoding drop 52.0% with an average accuracy drop of just 1.27 percentage points (87.01% to 85.74%); on Qwen-3.6-27B, 31.1% fewer attended tokens at a 2.75pp cost. The gap steadily narrows as scale grows from 4B to 31B, and a single response saves up to 21 million attended tokens. Ablations show the dynamic mask itself drives the bulk of the savings — up to 71.1% fewer attended tokens relative to the maskless ablation.
Using a roofline model (B200, bf16, 70% bandwidth utilization), the authors project decode wall-clock time of 0.71x on Gemma and 0.77x on Qwen on a well-optimized serving stack. The trade-off is more decode steps (+35% and +31% respectively), but with the global read consuming 73% (Gemma) and 86% (Qwen) of vanilla decode time, reading less beats stepping less.
How it differs from existing routes
Mainstream sparse attention pre-selects tokens via lightweight indexers or proxy scores, still paying an O(N) scan every step; resolving a <focus> tag is O(1). Compared with eviction-style compression that permanently drops KV entries, DA keeps the full cache resident and every per-step mask reversible. The authors call it a new axis of sparse attention — control moves from the system side into the model's own output stream.
Caveats worth stating
The paper is upfront about two hard limitations. First, all tested models fail to follow the DA protocol in thinking mode, so experiments disable thinking entirely — a direct friction with the reasoning-model mainstream. Second, zero-shot results are a lower bound; the protocol has not yet been post-trained into weights. The 0.71x figure is a modeled projection, not a production measurement, and shares like "global reads are 98.53% of decode at 1M context" shift quickly with context length.
So what: the most valuable part of DA is not the 52% but the demonstration that a model's knowledge of "what it is looking at" can be externalized into a parseable text protocol. If post-training bakes these declarations into weights — and vLLM integration is just a block-table hook — the cost accounting of long-context inference may genuinely need a rewrite.
Reference: arxiv.org/abs/2609.02737