[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-carve-gdn2-content-aware-recurrent":3,"news-related-f0cab9bc-1b73-4362-80c7-f621be56ef5c":36},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"f0cab9bc-1b73-4362-80c7-f621be56ef5c","CARVE 把 GDN-2 的「记忆盲区」补上：用输出张量「白嫖」内容信号，1.3B 模型长上下文检索刷新 SOTA","6 月 25 日挂上 arXiv 的 CARVE（Content-Aware Recurrent with Value Efficiency），把当下最被看好的「delta-rule 系」线性注意力架构 GDN-2 的三个耦合缺陷一锅端掉——代价是「白嫖」GPU 上本来就要写回 HBM 的输出张量，几乎不增加算力。\n\nGDN-2 被认为是 Transformer 之外最可能工业化的线性注意力路线：固定大小状态矩阵 S_t 把历史压成 d_v·d_k 个实数，每 token O(d_v·d_k) 更新，推理时延与序列长度彻底解耦。但 GDN-2 的 erase\u002Fwrite 门控「memory-blind」——只看新进 token，不看 S_t 已存什么。论文证明这种「value-axis erase mask」数学上破坏了 WY-form triangular chunk solver 的可行性，迫使 intra-chunk 求解退化为 d_v 个独立子问题，串行成本暴涨。\n\nCARVE 用一个原则同时解决三个问题：所有门控只放在 key axis 上。一旦 erase gate 与 value index 解耦，WY-form 单一三角求解立刻恢复，chunk 并行训练才有可能。在此约束下，作者做了两件工程化改造：\n\n- **Content-aware erase via output reuse**：erase gate 的「内容信号」直接复用本就要写回 HBM 的张量——上一 chunk recurrent output 的均值 m_c（one-chunk staleness）。这个零成本信号过零初始化低秩投影 U_b 后送到 gate；U_b=0 时 CARVE 与 GDN-2 bit-identical，训练中 gate 慢慢激活，0.18% 偏差在所有 chunk length 上都 flat——论文用定理证明扰动量级是 O(1\u002F√L)；\n- **Scalar value write gate**：把 per-value 的 d_v 维 write-gate 投影换成「每头一个标量」w_{h,t}。在 H=12, d_v=768 下，单层 write-gate 参数从 589,824 砍到 9,216，mixer 总参 −19%，峰值显存 −13%。\n\n效果由一组数据点支撑：1.3B \u002F 100B tokens \u002F FineWeb-Edu \u002F H100 三 seed 平均。WikiText 困惑度 15.72（vs 15.90，−0.18，跨 seed 4.5σ 效应），hybrid 变体 15.41；9 个常识推理基准全领先平均 +0.63 pp；RULER 上 S-NIAH 与 MK-NIAH 每一档 context length 都刷新 SOTA，6 项真实召回任务全 top。\n\n最关键的工程信号是「无硬件代价」：throughput 与 GDN-2 在 0.4% 测量噪声内持平。这意味着 CARVE 不是「更准但更慢」，而是「更准且更便宜」。当推理侧 KV-cache 已成为 LLM serving 真正的成本曲线（参考本月初 InfoKV、UltraQuant 一系列工作），能在不增加算力前提下把长上下文检索准确率再拉一档，对端侧 Agent、1M+ context RAG、多会话记忆系统都直接受益。\n\n论文还附 6 条形式化定理（记忆容量、Lyapunov 稳定性、梯度流、表达力分离、Pareto 最优 chunk size、hybrid 最优性），把 delta-rule 系的「工程经验」往「可证明的架构原则」推了一步——LLM 高效架构今年最值得收藏的「fix a real bug」式论文之一。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.27229","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"a61558e3-0c14-4818-b90e-2401fb116282","en","CARVE fixes GDN-2's memory blind spot for free","CARVE (Content-Aware Recurrent with Value Efficiency), posted to arXiv on June 25, fixes three coupled flaws in GDN-2 — currently the most-watched \"delta-rule family\" linear-attention architecture — at the cost of \"riding free\" on the output tensor that already has to be written back to HBM, with almost no extra compute.\n\nGDN-2 is considered the most likely linear-attention line to industrialize outside the Transformer: a fixed-size state matrix S_t compresses history into d_v·d_k real numbers, updates are O(d_v·d_k) per token, and inference latency is fully decoupled from sequence length. But GDN-2's erase\u002Fwrite gates are \"memory-blind\" — they look at the incoming token, not at what S_t already holds. The paper proves that this \"value-axis erase mask\" mathematically breaks the feasibility of the WY-form triangular chunk solver, forcing the intra-chunk solution to degenerate into d_v independent sub-problems, with serialized cost spiking.\n\nCARVE fixes all three with one principle: place all gates only on the key axis. Once the erase gate is decoupled from the value index, the WY-form single-triangular solver is immediately restored, and chunk-parallel training becomes possible. Within that constraint, the authors do two engineering refactors:\n\n- Content-aware erase via output reuse: the \"content signal\" for the erase gate directly reuses the tensor that already has to be written back to HBM — the mean m_c of the previous chunk's recurrent output (one-chunk staleness). This zero-cost signal passes through a zero-initialized low-rank projection U_b and is fed to the gate; at U_b=0 CARVE is bit-identical to GDN-2, and the gate activates slowly during training, with a 0.18% bias flat across all chunk lengths — the paper proves the perturbation magnitude is O(1\u002F√L).\n- Scalar value write gate: replace the per-value d_v-dimensional write-gate projection with \"one scalar per head\" w_{h,t}. At H=12, d_v=768, the write-gate parameters per layer drop from 589,824 to 9,216, the mixer's total params drop 19%, and peak memory drops 13%.\n\nA set of data points supports the effect: 1.3B \u002F 100B tokens \u002F FineWeb-Edu \u002F H100, three-seed average. WikiText perplexity 15.72 (vs 15.90, −0.18, 4.5σ effect across seeds); the hybrid variant 15.41; all nine commonsense-reasoning benchmarks led by an average +0.63 pp; every context length on RULER's S-NIAH and MK-NIAH refreshes SOTA; and all six real-recall tasks top the chart.\n\nThe most critical engineering signal is \"no hardware cost\": throughput is within 0.4% measurement noise of GDN-2. That means CARVE is not \"more accurate but slower\" — it's \"more accurate and cheaper.\" With KV cache already the true cost curve of LLM serving (cf. a wave of work like InfoKV and UltraQuant earlier this month), squeezing one more notch of long-context retrieval accuracy without raising compute directly benefits on-device Agents, 1M+ context RAG, and multi-session memory systems.\n\nThe paper also attaches six formal theorems (memory capacity, Lyapunov stability, gradient flow, expressive-power separation, Pareto-optimal chunk size, hybrid optimality), pushing the delta-rule family from \"engineering experience\" to \"provable architectural principle\" — one of this year's most \"fix a real bug\" papers worth keeping in the efficient-LLM-architecture drawer.","carve-gdn2-content-aware-recurrent","2026-06-29T04:00:00Z","2026-06-28T20:16:39.139424Z","2026-08-19T02:08:40.142862Z",true,"agent",92,{"items":37},[38,43,48,53,58,63],{"id":39,"title":40,"news_slug":41,"published_at":42},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"926f89fc-5ed5-4170-bfac-931d3a31b6a4","腾讯混元开源 AngelSpec 投机解码框架：DFly 在 Hy3-A21B 上取得 1.98–2.40× 加速","tencent-angelspec-spec-decoding-hy3-dfly","2026-07-30T00:00:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"070aef27-5fdb-4f0a-8b90-99afc1ea34fb","Jet-Long 用「动态双焦 RoPE」让 Qwen3 免训练扩到 128K,RULER 直接多涨 4.79 pp","jet-long-dynamic-dual-rope","2026-07-12T02:30:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"4cbfe2a4-5b83-464d-bc5b-50acf224b1b0","torch.profiler 实测 SDPA：FlashAttention 13% 占用率真相","pytorch-sdpa-flash-attention-13","2026-07-11T08:01:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"f63a58a9-85c9-406c-beef-0ba1cb0c6985","Taylor-Calibrate 把 Transformer 蒸馏成 GDN 的初始化做成系统级工程","taylor-calibrate-transformer-gdn-distill-88x","2026-06-21T14:30:00+00:00"]