Attention scores every query against every key with a full dot product. The standard way to make that cheaper is to show each query fewer keys: sliding windows, KV cache eviction, dynamic block selection. Hidden Sparse Attention, from EPFL and ETH Zurich, takes a different axis. It does not drop keys, it drops dimensions of the dot product itself. By the paper's own measurements, at 4K context it computes 29.3% of the query-key dot products and lands on a lower validation loss than dense attention.
1. RoPE makes the dot product decomposable
RoPE encodes position by rotating each two-dimensional frequency pair of the query and key. Rotations compose, so the query and key rotations collapse into one, and only the relative distance Delta = i - j survives in the inner product. The attention score decomposes exactly into a sum over frequency pairs.
High-frequency pairs complete many turns within a few tokens, so their score parts are extremely sensitive to the exact relative distance. Low-frequency pairs barely rotate across the whole context, and they carry the long-range dependency.
The HiSA rule is one sentence: the score part of pair l is kept only while the rotation it induces, Delta theta_l, stays below an angle budget Phi. The default is Phi = 90 degrees, meaning a pair contributes while RoPE has turned it by less than a quarter of a turn. The fastest pairs then see only a few tokens, the slowest several thousand, and together they define a reach. With d = 64 and Phi = 90, the reach is 11.8K tokens at RoPE base 10^4, and 23.1K at 2 times 10^4.
The sparsity pattern is static and parameter-free. It depends only on the distance and on the RoPE frequencies, not on the content of the queries and keys, so there is no router and no top-k selection, and nothing at runtime scores and picks keys. The only new quantity is the hyperparameter Phi; the projections, the values and the softmax are unchanged. Because the pattern is fixed by positions, the model is trained with exactly the pattern it uses at inference.
2. Less compute, lower loss
The authors pretrain SmolLM-architecture models of 135M and 360M parameters from scratch on OpenWebText, with a budget of about 20 tokens per parameter. At 4K context, with 135M and RoPE base 10^4, dense attention reaches a validation loss of 2.906 and HiSA 2.880, while computing 29.3% of the dot products. At base 2 times 10^4 the pair is 2.908 against 2.869, at 34.2% of the work. At 360M the numbers are 2.587 against 2.575, again at 29.3%. The gap holds at every position of the predicted token in the context window, so it is not an artifact of the first few tokens.
The kernels come in two flavours. The exact rule uses a hand-written Triton kernel and computes 29.3% at 4K, but trains 1.7 times slower than dense attention with SDPA. The block rule, Flash-HiSA, groups positions into blocks of 128 tokens and computes 34.8%, running at 238k tokens per second against 236k for dense FlashAttention-3. The two rules are different models: train and run a model with the same one.
Inference is where the saving actually shows up. At 16K context and batch 128, Flash-HiSA reaches 13.0k tokens per second with a KV cache of 29.1 GiB, against 9.2k tokens per second and 45.0 GiB for dense. That is 42% more throughput and a cache a third smaller. Even with the same kernel and no pair dropped at all, Flash-HiSA is still 13% faster, which the paper attributes to its smaller query tiles.
Existing dense models can be converted too. The dense 135M model is switched to HiSA without changing any weight, taking its validation loss from 2.906 to 2.937; after each continues for 1.35B tokens, dense reaches 2.838 and HiSA 2.836.
3. Dense attention is already suppressing those parts
The paper also asks whether dense models even use the frequency pairs that HiSA drops. The method compares a real score part against a control score part with the same rotation but unrelated content, and measures which one moves the attention output more. At initialization the two are close. After 1B tokens, only 2-5% of the cells below 90 degrees fall below the control, while beyond that line the real parts are gradually pushed down: in the 180 to 360 degree band the share grows from 33% to 48%, and in SmolLM-135M, trained on about 600B tokens, 87-95% of the cells beyond 180 degrees fall below the control. In terms of relative influence on head outputs, parts beyond 720 degrees drop from 4.6% after 1B tokens to 2.1% at the end of training, and to 0.5% in SmolLM-135M.
So a dense model spends part of its training suppressing score parts with a large rotation, and the suppression is still progressing at the end of the runs. HiSA writes that behaviour into the architecture.
4. Four cold-water notes
First, the experiments cover only 135M and 360M models. The paper itself lists scaling to bigger models, downstream task benchmarking and better kernel design as open questions.
Second, the exact rule trains 1.7 times slower than dense attention, so the speed argument depends on the block version, which is an approximation that keeps up to 127 more tokens per pair. The backward pass does not yet benefit from the sparsity, so only the forward pass saves anything.
Third, the paper trains a control variant that normalises each score by the number of pairs it keeps. It reaches a validation loss of 2.879, on par with HiSA at 2.880, so the gain does not come from the shrinking scale of distant scores. But the variant collapses on a long-range copy task, repeating a 32-token span seen 2560 tokens earlier: its token accuracy is 33%, against 77% for HiSA and 60% for dense attention. The dropped dimensions are not irrelevant on every task.
Fourth, at least two other papers were working on the same idea around the same time. ATFlash opens a distance window per RoPE wavelength and reports pruning 37-48% of the query-key inner-product terms on Qwen2.5-0.5B and Llama-3.2-3B, with the top-1 match rate staying at 96-98%. WavePrune restricts each channel to its first rotation period and reports the HELMET score on Qwen3-8B rising from 35.7 to 40.0, with 1.15x prefill and 1.24x decoding speedups over FlashAttention-2 at 32K context. The HiSA paper lists both, and frames its own differences in three ways: it trains with much tighter budgets, a quarter period at 90 degrees; it turns the rule into an actual KV-cache reduction; and it adds the interpretability analysis described above. The paper also notes that the other two skip computation and key reads but neither reduces the KV cache, since every key and value stays in memory.
5. So what
All of this points at one conclusion: the periodicity of RoPE is largely redundant at long range, because the distance over which a frequency pair can encode an exact relative position is finite. Three groups arrived at it from three directions, training loss, inference memory and output distribution.
What makes HiSA worth reading is that it turns this into an architecture rather than an inference trick. You can train a new model with it, or convert an existing dense model by continued training. For anyone serving long contexts, a KV cache roughly a third smaller is probably more interesting than the small difference in validation loss.
The caveat is that all of this comes from small models. Whether computing 30% of the dot product still holds on much larger models, in real long-document inference, has not been shown yet. The paper is at https://arxiv.org/abs/2611.26363 and the code repository at https://github.com/ambroise-b/hisa .