The bill for long context is easy to work out. Attention prefill cost grows quadratically with sequence length, and at decode time every new token forces a re-read of the entire KV cache. A 1M-token context is, in engineering terms, a simultaneous collision of memory capacity and bandwidth. Plenty of papers have worn the "subquadratic attention" label over the past two years, but very few actually achieve asymptotic subquadratic behavior — most merely shrink the constant factor, or relocate the cost from attention itself into the indexer.
A research team at Zyphra takes a different route in arXiv:2610.42112.
Attention was already sparse; nobody organized it
The first thing the paper does is not build a new operator. It measures a fact: on natural language sequences, the genuinely useful state of attention is sublinear. The team measured the size of the top-p KV subset in global attention layers of Qwen3.6-35B-A3B — the smallest set of keys whose attention probabilities sum to p — and found that across six long fiction books, that subset size grows as a sublinear power law with context length. Put differently: the longer the sequence, the smaller the fraction of tokens any single query actually needs to look at.
There is a trap here. A key that is irrelevant to the current query may matter enormously to a later one. On a 261,548-token sequence the team measured what fraction of the KV cache is covered by the union of top-p subsets across the W most recent queries, and found that union quickly approaches the entire cache. That directly rules out permanently evicting KV entries — the full cache has to stay, and each query simply retrieves its own subset.
A compressed tree instead of a flat indexer
The usual recipe for fine-grained sparse attention is simple: a lightweight indexer scores every KV pair, and exact attention runs only over the top-K. The problem is that the indexer itself may have to sweep the whole KV cache. The cost is moved rather than removed — at contexts approaching 1M tokens, 60% to 90% of the time in these layers is still spent on indexing.
ZHA builds a recursively compressed tree of key summaries on top of the KV cache. A query walks down from the top, each level routing to a candidate set of blocks via top-p selection and handing its children to the next level, until it lands on leaf blocks where exact attention runs. As the tree's depth grows iteratively, the total cost of routing plus final attention grows sublinearly in sequence length, yielding overall subquadratic (though still superlinear) scaling.
The key engineering property is that this structure is insensitive to geometry. The paper swept 28 configurations (reference query count R, chunk size C, tree depth L) and further swept routing thresholds from p = 0.7 to 0.99, finding retrieval accuracy fairly robust to tree shape when only 10% to 30% of keys are attended. At a fixed compression factor of C^L = 4096, going deeper actually lets effective retrieval hold up at higher sparsity. The team also fit RULER accuracy with a four-parameter model; leave-one-length-out extrapolation came out to an RMSE of just 1.44 percentage points.
Measured results: swap the indexer, leave the backbone alone
This is the most engineering-friendly aspect of ZHA: the backbone weights stay completely frozen, and only a router head is trained via KL distillation over a few billion tokens. On Qwen3.6-35B-A3B, the team trained both a "deep" router (chunk size 4, six tree levels) and a "wide" one (chunk size 256, one tree level), with routing thresholds of 0.8, 0.9 and 0.95.
The results: on reasoning, mathematical and agentic coding benchmarks, both routers matched dense attention within noise. The single exception was RULER-13 for the deep router — 93.3% against dense's 96.0%, a 3-point gap, and at that point only 1% to 6% of the original KV cache was being attended.
The comparison is the more interesting part. The baseline is MiniMax's sparse attention, MSA — blockwise sparsity with a lightweight Index Branch, which the authors report reduces per-token attention compute by 28.4x at 1M context on a 109B model, with 14.2x prefill and 7.6x decoding wall-clock speedups on H800. The team gave MSA's indexer a 40B-token distillation warmup, 10x ZHA's training budget. But on LongBench v2 — where MSA's fixed budget amounts to roughly 1% of the context — the ZHA deep router attends a comparable 1.1% of decode keys and still scores 14 points higher.
The same method was also ported onto Nemotron-3.5-Lightning-30B-A3B, which uses NoPE and carries no explicit positional encoding. Agentic coding performance is broadly comparable, while RULER degrades moderately. Interestingly, the top-p subsets of the NoPE model do not exhibit such a regular power law — the paper credits that to the positional encoding itself.
The "million-token context" is extrapolated, not measured
This is the point in the paper that most needs flagging. The authors extrapolate ZHA's latency scaling with sequence length using a roofline model, concluding that beyond 1M tokens it would "dramatically outperform" both dense attention and MSA, with a predicted crossover against MSA around 0.5M tokens at p = 0.75.
The paper also states plainly that, for lack of LLMs able to operate at such lengths, these ultra-long scalings have not been empirically validated, and that this work does not release optimized kernels. So "subquadratic" currently amounts to a theoretically grounded model prediction plus a set of measurements between 64K and 256K; the extrapolated part will only be cashed out once hardware and models advance together. One self-referential observation the authors make is that, to their knowledge, this is the first hierarchical attention scheme to demonstrate convincing scaling to depths greater than 1 while staying close to dense attention — and that "first" currently rests on the paper's own assertion.
So what
What is worth remembering is not "yet another subquadratic attention," but three reusable judgments.
First, attention is already sublinearly compressible on natural language — once that is measured, "not enough context" starts to look like an engineering bill rather than a capability ceiling. Second, the indexer has to be a hierarchy, not a flat top-K scoring pass, if savings are to keep compounding with length; otherwise the attention cost you saved comes back in the indexer. Third, the migration cost is remarkably low: frozen backbone, 4B tokens, a few percent of attention exposure, and you land close to dense performance.
For people building long-context agents, the third point is probably the most immediately useful. You do not have to wait for a new model release, and you do not have to retrain — a few percent of the attention budget can put an entire codebase or long document into the window. As for the vision of hundreds of millions of tokens, treat it as a roadmap with a theoretical foundation, not as a number that has already been demonstrated.