[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-attention-3-1-hybrid-qwen-kimi-deltanet":3,"news-related-8f7e0745-4bb1-4226-8d6e-c9d159e6bf78":36},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"8f7e0745-4bb1-4226-8d6e-c9d159e6bf78","注意力架构收敛：3:1混合比成为2026年大模型新范式","2026年第一季度，两家独立团队——阿里巴巴Qwen团队和Moonshot AI（月之暗面）——在没有相互参考的情况下，得出了一致的工程结论：在线性注意力与全注意力之间，3:1的混合比例是最优解。\n\nQwen3.5于2026年2月发布，采用Gated DeltaNet线性注意力机制，以3层线性注意力搭配1层标准注意力的混合结构运行；几乎同期，Kimi Linear基于自主研发的KDA（Kimi Delta Attention）机制，实现了完全相同的3:1混合配比。这不是巧合，而是架构层面的深度收敛。\n\n为什么这个比例如此重要？传统Transformer的全注意力计算复杂度为O(n²)，上下文翻倍，计算量增加四倍。线性注意力将复杂度降至O(n)，但纯线性方案在表达能力上存在损失。3:1混合在效率和能力之间找到了平衡点：KV Cache内存降低75%，百万token上下文下吞吐量提升6倍，同时保留足够的全注意力层来维持模型的核心表达能力。\n\n这场架构收敛背后有一个被低估的推动者——NVIDIA Research。Gated DeltaNet最初由NVIDIA团队在ICLR 2025上提出，本质上是在Mamba2状态空间模型基础上引入门控机制，从而更高效地利用有限状态记忆。如今Qwen、Kimi等头部团队均基于这一架构进行研发，NVIDIA实际上掌握着这场范式转换的底层定义权。\n\n对行业而言，混合注意力比成为标准，意味着长上下文推理的成本结构将发生根本性变化。百万token上下文的工程可行性大幅提升，Ring Attention等分布式方案与混合注意力的结合，可能让超长上下文从技术展示走向工业级应用。下一步的真正问题是：这套架构能否在保持效率优势的同时，进一步缩小与纯全注意力模型在复杂推理任务上的差距。","https:\u002F\u002Fsebastianraschka.com\u002Fllms-from-scratch\u002Fch04\u002F08_deltanet\u002F","8c758013-1efc-4f1d-bc10-8860362115e7",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":18,"name":19,"slug":19,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":21,"name":22,"slug":22,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"127e97dd-615a-4836-ae79-1903c358d5b8","en","Attention converges: the 3:1 hybrid ratio becomes 2026's LLM norm","In Q1 2026, two independent teams — Alibaba's Qwen team and Moonshot AI — without referencing each other, arrived at the same engineering conclusion: between linear and full attention, the 3:1 hybrid ratio is the optimal solution.\n\nQwen3.5, released in February 2026, adopts the Gated DeltaNet linear attention mechanism, running with a 3-layer linear attention paired with 1-layer standard attention hybrid structure; around the same time, Kimi Linear, based on the in-house KDA (Kimi Delta Attention) mechanism, achieved exactly the same 3:1 hybrid ratio. This is not coincidence, but deep architectural convergence.\n\nWhy is this ratio so important? Traditional Transformer's full attention compute complexity is O(n²), and doubling the context quadruples compute. Linear attention reduces complexity to O(n), but pure linear solutions lose expressiveness. The 3:1 hybrid finds a balance point between efficiency and capability: KV Cache memory reduced by 75%, throughput up 6× under million-token context, while retaining enough full-attention layers to maintain the model's core expressive capability.\n\nBehind this architectural convergence is an underestimated driver — NVIDIA Research. Gated DeltaNet was originally proposed by the NVIDIA team at ICLR 2025, essentially introducing a gating mechanism on top of the Mamba2 state-space model, thereby more efficiently using limited state memory. Today, top teams including Qwen and Kimi are doing R&D based on this architecture, and NVIDIA effectively holds the underlying definition rights for this paradigm shift.\n\nFor the industry, the hybrid-attention ratio becoming standard means the cost structure of long-context inference will undergo fundamental change. The engineering feasibility of million-token context is greatly improved; the combination of distributed solutions like Ring Attention with hybrid attention may bring ultra-long context from tech showcase to industrial-grade application. The real next question is: while maintaining efficiency advantages, can this architecture further close the gap with pure full-attention models on complex reasoning tasks.","attention-3-1-hybrid-qwen-kimi-deltanet","2026-05-03T13:05:00Z","2026-05-03T13:08:32.507283Z","2026-08-19T02:08:40.142862Z",true,"agent",572,{"items":37},[38,43,48,53,58,63],{"id":39,"title":40,"news_slug":41,"published_at":42},"217f417d-1b9c-475b-99f4-e21e7c909711","MHAR 把 Transformer 残差流从「单车道」拆成 H 条独立路由:子空间第一次有权自己挑历史层","multi-head-attention-residuals-mhar","2026-08-01T07:30:00+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"623f7e16-ef9a-43fc-9303-d01bfd60d8fe","把 LLM 推理拆成四层架构：62 页综述给「Token 运营」补一条产业视角","token-operations-four-layer-62-page-survey","2026-06-18T14:33:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"4e43e35d-a808-4125-be31-69cadedc61f1","PoLar 把 LLM 层变成可调积木：动态跳层+复读，3B 模型数学推理涨 60+ 个百分点","polar-icml-2026-3b-math-62pp-jump","2026-06-15T14:00:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"fc93d022-8522-4396-a047-c9ba8fc1821c","VIA-SD 入选 ICML 2026：投机解码终于有了「瘦验证器」，推理再快 20%","via-sd-icml-2026-slim-verifier-20pct","2026-06-11T20:15:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"3a9a8c69-d668-4d2c-ae82-caeba45aa2d5","MIT新方法利用计算空闲周期：推理模型训练速度翻倍，能耗减半","mit-rllm-idle-cycle-2x-train-half-energy","2026-05-22T08:10:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"2e1d1723-4cea-4621-965e-9514d08a9013","LLM推理服务正在淘汰「启发式」：运筹学视角下的新优化范式","llm-inference-or-paradigm-heuristics","2026-05-16T08:25:00+00:00"]