Linear attention attacks a simple problem: softmax attention scales quadratically with sequence length during training and keeps a linearly growing cache at inference. Compressing context into a fixed-size key-value state buys linear-time training and constant-memory inference — the shared pitch of the gated delta-rule family, from Gated DeltaNet through Kimi Linear to GDN-2, which generalizes the line with channel-wise decay and decoupled full-rank erase/write gates. On November 5, three Google researchers — Tushaar Gangavarapu, Gil Tabak, and Clemens JS Schaefer — posted Muir to arXiv. The work is essentially a parameter-budget audit: first prove that the erase gates in this architecture are over-parameterized, then reinvest the savings where they actually pay.
Two overlooked observations
The paper leads with analysis rather than architecture. First, state updates are far less sensitive to erasure than to writes: key activation and unit normalization concentrate key energy onto a sparse subset of channels, while value writes scale all channels — so the erase gate can safely be parameterized at lower rank. Second, slow-decaying state rows keep accumulating value writes over hundreds to thousands of tokens, reaching norms more than an order of magnitude larger than fast rows. Under unweighted readout these high-norm slow rows dominate the output, forcing stronger write-time erasure to suppress interference — which prematurely overwrites the very associations the state was supposed to keep.
How Muir spends the budget
Guided by that analysis, Muir makes three reallocations. It fuses all recurrence and output gates into a sensitivity-allocated low-rank projection. It re-weights state rows with a matched-filter readout gate that combines a static channel profile with dynamic per-token modulation. And it reinvests the saved parameters into a wider feed-forward network. The accounting is concrete: in GDN-2, two dense d×d gate projections consume 2d² parameters — one-sixth of the 12d² per-layer budget — squeezing FFN width by over 25% under iso-parameter constraints. Muir widens the FFN by 30.4% (from 5,888 to 7,680) while cutting compiled per-step FLOPs by 1.8% and HBM traffic by 0.7–1.4%. Since the FFN is where factual associations live, the reinvestment converts directly into memory capacity.
The numbers
Across 166M–1.34B parameters, Muir lowers perplexity by 1.8–2.0% over GDN-2 and 2.5–3.6% over Transformer, and raises commonsense accuracy by up to 1.9 points. At 1.34B it beats both baselines on multi-hop and multi-needle recall, extrapolates 2,560× from its 4K training length to 10M-token streams without perplexity drift, and keeps near-100% passkey retrieval at 1M tokens with static readout — whereas GDN-2 collapses past 4K–8K. The engineering side is just as sharp: Muir matches GDN-2 throughput, reaches its 100B-token validation loss 1.2× faster, and trains 2.7× faster than Transformer at 32K.
The kernel behind this is a boundary-pivoted chunkwise update designed for wide matrix units. On the 88 TPU7x Ironwood chips used in the paper, each chip pairs two TensorCores — 256×256 systolic arrays delivering 2,307 TFLOP/s in bfloat16 — with a vector unit roughly 64× slower. Standard chunkwise algorithms divide keys by cumulative decay prefixes, which underflows on fast-decaying channels at the large chunk sizes these accelerators need. Muir's kernel splits relative decays without division and solves within-chunk triangular systems via block matrix multiplications, keeping 98% of FLOPs on the matrix units.
Keep it cool
The paper is candid about limits. Transformer remains faster at short contexts (up to 8K), and Splash Attention wins below roughly 11K, partly because the pure-JAX implementation pays XLA kernel-boundary HBM traffic between chunk stages. Static readout cannot match varying lags on multi-hop and extractive tasks — dynamic readout is what holds perplexity flat across 10M-token streams — and the authors propose pairing sliding-window attention with static-readout Muir as a hybrid. No weights or code have been released: the Hugging Face paper page lists zero linked models, and 1.34B is far from frontier scale.
The takeaway: linear-attention competition is shifting from "replace the architecture" to "audit every parameter." Under a fixed budget, whoever spends gate capacity where it matters gets orders-of-magnitude long-context extrapolation nearly for free. With Kimi Linear, GDN-2, and Muir iterating along one family tree, this path is far from converged. The paper is at arxiv.org/abs/2611.08376.