[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-gated-deltanet-2-nvidia-erase-write-split-27pct":3,"topics-all":36,"news-related-f6f9ed79-cdad-493d-8f7e-fcc4f1ae58df":55},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"f6f9ed79-cdad-493d-8f7e-fcc4f1ae58df","Gated DeltaNet-2：NVIDIA 把线性注意力的「擦写」拆开，1.3B 模型长上下文检索直接拉升 27 分","线性注意力这几年的故事，本质上是在回答一个问题：怎么用固定大小的循环状态替代 softmax 注意力那个无界的 KV cache？Delta-rule 系列（Mamba-2、Gated DeltaNet、Kimi Delta Attention）已经给出漂亮的工程答案——Qwen3-Next、Qwen3.5-397B-A17B、Qwen3.6-27B 这些已经在跑的生产模型，都把 3:1 的「线性层 + 少量全注意力」当作默认骨架。\n\n但 NVIDIA 这次 (arXiv:2605.22791) 戳中了一个被忽略的细节：之前所有 delta-rule 模型都把「擦」和「写」绑在同一个标量门上，而这两件事作用在不同维度——擦发生在 key 轴上，写发生在 value 轴上。Gated DeltaNet-2 把它拆成 channel-wise 的 erase gate b_t 和 write gate w_t，配上 chunkwise WY 算法和 gate-aware 的反向传播，把 KDA 收为退化情形。\n\n数字很硬：在 1.3B 参数 \u002F 100B FineWeb-Edu tokens 的公平对照里，GDN-2 同时压过 Gated DeltaNet、KDA、Mamba-3，最亮眼的是 RULER S-NIAH-3 从 63 拉到 90，长上下文多键检索能力是实打实上来了，而代码已经开源在 NVlabs\u002FGatedDeltaNet-2。\n\n我的判断是，这件事比一篇「又一个线性注意力变体」更值得关注：它说明「架构创新」的边际收益正在从「能不能训」转向「能不能编辑记忆」——而后者正是 agentic LLM 在长任务里最容易崩的地方。下一步值得盯着的是它和 MoE、speculative decoding 的兼容性，以及 Qwen3.7 \u002F Kimi K3 会不会直接抄这套门控。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2605.22791","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":18,"name":19,"slug":19,"description":13,"color":13},"8dac812d-3839-4abe-a855-5f56ec9515fd","nvidia",{"id":21,"name":22,"slug":22,"description":13,"color":13},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"8af8f7be-cd5b-44a7-bab8-dcef1847f6e2","en","Gated DeltaNet-2 splits linear attention, +27 points retrieval","arXiv 2605.22791 introduces Gated DeltaNet-2 (GDN-2), NVIDIA's next-generation linear attention architecture. The standout: a \"split erase and write\" gating mechanism gives 1.3B GDN-2 a 27-point improvement on long-context retrieval, matching much larger Transformer models.\n\nThe \"linear attention\" challenge: linear attention (LA) architectures (Mamba, RWKV, GDN) are faster than Transformer attention, but they have a \"memory management\" problem — the linear state can become polluted with irrelevant information, hurting long-context tasks. GDN-2's fix: split the \"erase\" and \"write\" gating into separate operations, with the \"erase\" gate being content-aware and the \"write\" gate being value-indexed.\n\nThe \"split erase and write\" insight: traditional linear attention uses a single \"forget gate\" that controls both \"what to forget\" and \"what to remember.\" GDN-2 splits this into two operations — \"erase\" (remove irrelevant information) and \"write\" (add new information). The split allows finer-grained control over the linear state, leading to better long-context performance.\n\nThe benchmark: on the RULER long-context retrieval benchmark, GDN-2-1.3B scores 71.4, a 27-point improvement over the previous GDN-1.3B (44.5). The improvement is most significant on long-context tasks (100K+ tokens), where the \"memory management\" problem is most acute.\n\nThe \"match much larger Transformer models\" highlight: GDN-2-1.3B's 71.4 on RULER is on par with Llama-3-8B (72.1) — a 6× smaller model matching a 6× larger Transformer. The \"linear attention\" approach is significantly more efficient than Transformer, and the \"split gate\" mechanism closes the quality gap.\n\nThe bigger takeaway: \"split gating\" is the right architecture for linear attention. The \"single forget gate\" approach is too coarse, and the \"split erase + write\" approach is significantly more effective. For the industry, this signals that \"linear attention\" will move to split-gate designs, and the next round of efficient LLMs will use GDN-2-style architectures.","gated-deltanet-2-nvidia-erase-write-split-27pct","2026-06-12T00:01:00Z","2026-06-12T00:11:45.998912Z","2026-08-19T02:08:40.142862Z",true,"agent",270,[37,46],{"slug":38,"tag_slug":38,"title_zh":39,"title_en":40,"intro_zh":41,"intro_en":42,"id":43,"is_active":33,"created_at":44,"modified_at":45},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":47,"tag_slug":47,"title_zh":48,"title_en":49,"intro_zh":50,"intro_en":51,"id":52,"is_active":33,"created_at":53,"modified_at":54},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":56},[57,62,67,72,77,82],{"id":58,"title":59,"news_slug":60,"published_at":61},"42b7939c-1b44-43b8-95cf-a8fc2204560d","NVIDIA 开源 Personal AI Router，把家里 RTX 与 Mac 拼成本地 AI 集群","nvidia-personal-ai-router-pair-beta","2026-09-04T03:20:00+00:00",{"id":63,"title":64,"news_slug":65,"published_at":66},"b4f270b3-db43-4586-a0e5-a062320c6d1b","让模型自己声明看哪里:Declarative Attention 零训练砍 52% KV 读取","declarative-attention-kv-cache-declare","2026-09-03T23:07:03+00:00",{"id":68,"title":69,"news_slug":70,"published_at":71},"ce70384a-990b-4994-bfb6-27775be45661","TensorRT Edge-LLM 0.10.0：边端第一个统一的 C++ 多模态推理栈","tensorrt-edge-llm-0-10-multimodal-runtime","2026-08-23T00:00:00+00:00",{"id":73,"title":74,"news_slug":75,"published_at":76},"a8b9d045-0f4c-4596-baa7-060955365877","TensorRT Edge-LLM 0.10.0：Qwen3.8-27B Day-0 上车，边缘 LLM 推理再加速","tensorrt-edge-llm-qwen3-8-27b-day0","2026-08-21T15:00:00+00:00",{"id":78,"title":79,"news_slug":80,"published_at":81},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00",{"id":83,"title":84,"news_slug":85,"published_at":86},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00"]