Hybrid LLM architectures were supposed to be the answer to long-context cost: replace most full-attention layers with linear attention, and per-step cost stops growing with context. But the few full-attention layers that remain still read the entire KV at every step, and at 256k tokens they become the new decoding bottleneck. A paper posted to arXiv on November 4 offers a fix that touches no weights: no retraining, no parameter edits — just a per-layer search over sparse-attention configurations. The result is a 2.23x decoding speedup at 256k context on Qwen3.5-122B-A10B, with accuracy intact across 21 long-context datasets.

The remaining full-attention layers are the long-context bill

Standard self-attention makes every generated token more expensive than the previous one: quadratic time in sequence length, and a KV cache that grows with every token and every concurrent request. Hybrid models swap most layers for cheap attention, but how many full-attention layers survive is frozen into the weights at pretraining: Qwen3.5-27B keeps 16 out of 64, gpt-oss-120b keeps 18 out of 36, and Ornith-397B keeps 15 out of 60. Those layers read every key in the context at each step; they eat most of the decode budget on long tasks and cap how many concurrent requests a deployment can serve.

The method: sliding window + prefix + query-aware pages, searched per layer

The paper's combined sparse attention lets each full-attention layer see only three things: a sliding window of recent tokens, a pinned prefix, and a fixed budget of the most relevant tokens selected dynamically per query — Quest-style paged selection over the KV cache.

The real insight is "per layer". Window size and token budget are not hand-picked: an NSGA-II multi-objective search (accuracy vs estimated cost) finds a (window, budget) pair for every layer individually, with accuracy estimated on BABILong-qa1 at 64k plus the first 100 LongBench Qasper examples. The search is not free but bounded: 8.6–34.2 GPU-hours on A100s for the 4B–120B models, and 31.3–58.4 GPU-hours on a 4x GB200 node for the 122B and 397B models. The implementation plugs directly into vLLM.

Two honest caveats: prefill is untouched, and KV-cache memory is not reduced at all — the savings are in keys read per decoding step, that is, time, not VRAM.

Results and limits: a free 2x, and where you don't get it

The evaluation covers six models — Qwen3.5-4B/27B/122B-A10B, Ornith-1.0-397B, Gemma-4-31B, gpt-oss-120b — across 21 long-context datasets (BABILong qa1–qa5 at 64k plus the 16 English LongBench subsets), with the samples used during search excluded so all reported accuracy is held-out.

On every model, at least one configuration preserves baseline accuracy on every dataset, with averages within 0.5 percentage points. At 64k/256k context, batch size 4: 1.26x/2.22x on Qwen3.5-4B, 1.20x/2.23x on Qwen3.5-122B-A10B, and 1.16x/2.06x on Ornith-1.0-397B — the three best models all double their decoding speed at 256k.

The limits are stated plainly: at short context, every configuration is slower than the unmodified model. Dense models gain little — Qwen3.5-27B gets only 1.11x at 64k and Gemma-4-31B 1.22x at 128k, because at 64k the 27B model spends 19.4ms of its 47.1ms decode step reading weights, which attention tuning cannot fix. gpt-oss-120b manages only 1.16x at 128k, since the searched configuration kept most of its full-attention layers intact.

Why it matters

Compare the retraining route: LoLCATs spent 454 H100 GPU-hours transferring attention on Llama 3.1 405B; Jet-Nemotron-2B hit a 12x speedup at 64k but paid with 600B training tokens and 18,328 H100 GPU-hours. A config search priced in GPU-hours means you can re-run it for every newly released model — a completely different amortization logic.

One more telling detail: the searched configs are not uniform. Ornith-397B's best-accuracy configuration leaves two of its 15 full-attention layers completely untouched, with per-layer windows ranging from 256 to 4096 — "how much context each layer needs" turns out to be a search result, not an architect's intuition.

The authors also state their own claim: "To our knowledge, this is the first training-free sparse-attention method evaluated at the 100B+ scale." Treat it as a self-report, not a settled fact.

Full paper: arXiv:2611.06133. Next time you plan long-context capacity, ask each layer: do you really need to read all 256k?