arXiv 2605.22791 introduces Gated DeltaNet-2 (GDN-2), NVIDIA's next-generation linear attention architecture. The standout: a "split erase and write" gating mechanism gives 1.3B GDN-2 a 27-point improvement on long-context retrieval, matching much larger Transformer models.
The "linear attention" challenge: linear attention (LA) architectures (Mamba, RWKV, GDN) are faster than Transformer attention, but they have a "memory management" problem — the linear state can become polluted with irrelevant information, hurting long-context tasks. GDN-2's fix: split the "erase" and "write" gating into separate operations, with the "erase" gate being content-aware and the "write" gate being value-indexed.
The "split erase and write" insight: traditional linear attention uses a single "forget gate" that controls both "what to forget" and "what to remember." GDN-2 splits this into two operations — "erase" (remove irrelevant information) and "write" (add new information). The split allows finer-grained control over the linear state, leading to better long-context performance.
The benchmark: on the RULER long-context retrieval benchmark, GDN-2-1.3B scores 71.4, a 27-point improvement over the previous GDN-1.3B (44.5). The improvement is most significant on long-context tasks (100K+ tokens), where the "memory management" problem is most acute.
The "match much larger Transformer models" highlight: GDN-2-1.3B's 71.4 on RULER is on par with Llama-3-8B (72.1) — a 6× smaller model matching a 6× larger Transformer. The "linear attention" approach is significantly more efficient than Transformer, and the "split gate" mechanism closes the quality gap.
The bigger takeaway: "split gating" is the right architecture for linear attention. The "single forget gate" approach is too coarse, and the "split erase + write" approach is significantly more effective. For the industry, this signals that "linear attention" will move to split-gate designs, and the next round of efficient LLMs will use GDN-2-style architectures.