Finding out what an attention head actually does normally means running data: activation patching, probing classifiers, crafted inputs — all of it needs forward passes. An arXiv paper (2611.12206) by Miguel P. Bento takes the opposite route: treat attention as a geometric object with continuous symmetries, compute a single scalar ρ straight from the weights, and characterize heads without a single forward pass or token of data. The paper was accepted as an extended abstract at the Symmetry and Geometry in Neural Representations Workshop (NeurReps), NeurIPS 2026.
What ρ measures
In RoPE models the attention logit splits into a static content term G_QK(0) = W_Q^T W_K plus a positional perturbation P(Δ) = W_Q^T [R(Δ) − I] W_K, where R(Δ) is the rotation at relative position Δ. ρ is simply the average of ||P(Δ)||²_F over ||G_QK(0)||²_F across a logarithmically spaced set of offsets — how big the positional term is relative to the content term.
Two properties matter. First, ρ is basis-invariant: G_QK is unchanged under the whole family of (W_Q, W_K) reparameterizations, and ρ inherits that invariance plus scale invariance. Second, it measures capacity, not usage: a head with ρ = 0 produces position-independent logits for any input (Proposition 1), while a high-ρ head merely has the capacity to couple to position — it need not do so on a given input. One caveat the authors note: ρ is sensitive to softmax shifts, though weight decay tends to suppress the offending components in practice.
Induction heads hide in the low-ρ tail
Induction heads implement the "A B … A → B" copying pattern, the most-studied circuit in mechanistic interpretability. Content matching is position-invariant by nature, so the prediction is that induction heads have extremely low ρ. Four bias-free RoPE models confirm it: SmolLM2-135M, TinyLlama-1.1B, SmolLM2-1.7B and Mistral-7B show Spearman correlations between ρ and induction scores from -0.55 to -0.74, all p < 10⁻⁴⁰, and the ten strongest induction heads always sit in the bottom 7%-30% of the ρ distribution.
The correlation is imperfect, and the authors say why: induction is a multi-head circuit, so low ρ is necessary but not sufficient — which is exactly the capacity interpretation.
Gating YaRN with ρ
Uniform RoPE scaling hurts short-context performance because it modifies every head indiscriminately. The authors instead apply YaRN only to the top-50% of heads by ρ, leaving the rest at their original positions. On Mistral-7B, evaluated within the native context window: at 512 tokens, perplexity is 7.63 for the baseline, 19.37 for uniform YaRN and 12.92 for the targeted variant; at 4096 tokens, 5.44, 162.96 and 130.34 respectively. The authors explicitly call this a preliminary proof of concept.
So what
The boundaries are stated plainly: RoPE architectures only, the QK sector only, models up to 7B, and the YaRN experiment on a single model. The authors also sketch where this generalizes: the G_QK eigenspectrum, the singular values of G_VO, and the commutator between the two could form an invariant signature per head — a data-independent counterpart to activation-based circuit maps.
The interesting part is the cost structure. Weights are static, cheap and reproducible; activation analysis is expensive and data-dependent. If invariants keep holding at larger scales, interpretability gains a cheap "screen first, inspect later" step. Code is on GitHub (github.com/MPedraBento/rho-invariant); the paper is at arxiv.org/abs/2611.12206. When reading a model's weights is enough to read its heads, which head would you check first?