In the long-context arms race, linear attention is the budget route: it flattens the quadratic cost of softmax attention, at the price of turning memory into a matrix that keeps being rewritten. The question is how you rewrite it — how much old memory gets disturbed each time you write something new. A new paper from the NX-AI team, KronLSTM (arXiv:2610.26834, submitted October 19, six authors including Sepp Hochreiter), offers a more surgical answer.
Write-in Interference: The Old Problem of Linear Attention
The repository README lines up three write behaviors in one animation: in classic mLSTM, every new value written disturbs all stored values in the matrix; in Gated DeltaNet, old values sharing dimensions with the new key get erased in bulk; KronLSTM instead modifies only the small block that overlaps with the new key-value pair, leaving the rest of memory intact. For long contexts this is not a detail — the longer the context, the more writes, and the harder it is for repeatedly disturbed old memories to stay faithful.
The Kronecker Delta Rule: Touch Only What Must Change
The core contribution is a new update rule for associative matrix memories. KronLSTM extends the xLSTM family with a memory-revision mechanism: when correcting a targeted association, it modifies memory along a single direction in matrix space, reducing interference with previously stored information compared to delta-rule updates. In theory, the resulting memory dynamics can be described as a Markov chain converging to a steady-state distribution, and the expected norm of the memory matrix stays uniformly bounded across the entire input sequence — meaning no extra normalization or stabilization machinery is needed for the memory state; it is stable by construction. On the experimental side, the paper's own wording: KronLSTM is competitive with, and often outperforms, existing linear attention architectures including the (Gated) DeltaNet family, with particularly strong performance on long-context benchmarks.
The Open-Source Package: FLA-Compatible, Four 1.5B Models
The engineering side is thorough. The code ships as Triton kernels plus a PyTorch layer, directly compatible with the flash-linear-attention (FLA) ecosystem: training uses chunk_kronlstm (a chunkwise WY/UT form, the same recipe as GDN-2 and KDA), decoding uses fused_recurrent_kronlstm. The repository is itself a snapshot of FLA with KronLSTM added in FLA's layout, and the authors say it is intended to be merged into FLA upstream. Four pretrained models are released, all 1.5B parameters, trained on 100B tokens with the Llama-2 tokenizer: two pure KronLSTM variants (24 layers, 8K and 32K context), and two hybrids (20 KronLSTM layers + 4 sliding-window attention layers, window 2048, no positional encoding). The training recipe is disclosed in detail: the 8K models train on FineWeb-Edu; the 32K variants branch from the 8K runs at the 75B-token mark and train the remaining 25B tokens at 32K context on a long-document mixture from ProLong-64k. The license is MIT; development and testing were done on NVIDIA H100 and A100 GPUs with PyTorch 2.13 and Triton 3.7.1.
So What
Two takeaways. First, linear attention competition has moved from "can we make it linear" into the deep water of memory-revision precision: NVIDIA's Gated DeltaNet-2 in June split erase from write, and the Kronecker Delta Rule goes one step further by restricting even the correction to a single direction — architectural micro-innovation is making a comeback, and whoever's memory comes closest to "touch only what must change" wins the long-context standoff. Second, for practitioners this is plug-and-play: FLA users can swap in the layer, and the authors have teased faster kernels on the way in the HF community discussion. A dose of realism: 1.5B parameters and 100B tokens is a research-scale configuration, and "often outperforms" is the paper's self-reported wording — production-grade validation is still ahead. Still, it gives the long-context track a new baseline worth watching.
Reference: arXiv:2610.26834 (abs page); github.com/NX-AI/KronLSTM; huggingface.co/collections/NX-AI/kronlstm