CARVE (Content-Aware Recurrent with Value Efficiency), posted to arXiv on June 25, fixes three coupled flaws in GDN-2 — currently the most-watched "delta-rule family" linear-attention architecture — at the cost of "riding free" on the output tensor that already has to be written back to HBM, with almost no extra compute.

GDN-2 is considered the most likely linear-attention line to industrialize outside the Transformer: a fixed-size state matrix S_t compresses history into d_v·d_k real numbers, updates are O(d_v·d_k) per token, and inference latency is fully decoupled from sequence length. But GDN-2's erase/write gates are "memory-blind" — they look at the incoming token, not at what S_t already holds. The paper proves that this "value-axis erase mask" mathematically breaks the feasibility of the WY-form triangular chunk solver, forcing the intra-chunk solution to degenerate into d_v independent sub-problems, with serialized cost spiking.

CARVE fixes all three with one principle: place all gates only on the key axis. Once the erase gate is decoupled from the value index, the WY-form single-triangular solver is immediately restored, and chunk-parallel training becomes possible. Within that constraint, the authors do two engineering refactors:

  • Content-aware erase via output reuse: the "content signal" for the erase gate directly reuses the tensor that already has to be written back to HBM — the mean m_c of the previous chunk's recurrent output (one-chunk staleness). This zero-cost signal passes through a zero-initialized low-rank projection U_b and is fed to the gate; at U_b=0 CARVE is bit-identical to GDN-2, and the gate activates slowly during training, with a 0.18% bias flat across all chunk lengths — the paper proves the perturbation magnitude is O(1/√L).
  • Scalar value write gate: replace the per-value d_v-dimensional write-gate projection with "one scalar per head" w_{h,t}. At H=12, d_v=768, the write-gate parameters per layer drop from 589,824 to 9,216, the mixer's total params drop 19%, and peak memory drops 13%.

A set of data points supports the effect: 1.3B / 100B tokens / FineWeb-Edu / H100, three-seed average. WikiText perplexity 15.72 (vs 15.90, −0.18, 4.5σ effect across seeds); the hybrid variant 15.41; all nine commonsense-reasoning benchmarks led by an average +0.63 pp; every context length on RULER's S-NIAH and MK-NIAH refreshes SOTA; and all six real-recall tasks top the chart.

The most critical engineering signal is "no hardware cost": throughput is within 0.4% measurement noise of GDN-2. That means CARVE is not "more accurate but slower" — it's "more accurate and cheaper." With KV cache already the true cost curve of LLM serving (cf. a wave of work like InfoKV and UltraQuant earlier this month), squeezing one more notch of long-context retrieval accuracy without raising compute directly benefits on-device Agents, 1M+ context RAG, and multi-session memory systems.

The paper also attaches six formal theorems (memory capacity, Lyapunov stability, gradient flow, expressive-power separation, Pareto-optimal chunk size, hybrid optimality), pushing the delta-rule family from "engineering experience" to "provable architectural principle" — one of this year's most "fix a real bug" papers worth keeping in the efficient-LLM-architecture drawer.