In Q1 2026, two independent teams — Alibaba's Qwen team and Moonshot AI — without referencing each other, arrived at the same engineering conclusion: between linear and full attention, the 3:1 hybrid ratio is the optimal solution.
Qwen3.5, released in February 2026, adopts the Gated DeltaNet linear attention mechanism, running with a 3-layer linear attention paired with 1-layer standard attention hybrid structure; around the same time, Kimi Linear, based on the in-house KDA (Kimi Delta Attention) mechanism, achieved exactly the same 3:1 hybrid ratio. This is not coincidence, but deep architectural convergence.
Why is this ratio so important? Traditional Transformer's full attention compute complexity is O(n²), and doubling the context quadruples compute. Linear attention reduces complexity to O(n), but pure linear solutions lose expressiveness. The 3:1 hybrid finds a balance point between efficiency and capability: KV Cache memory reduced by 75%, throughput up 6× under million-token context, while retaining enough full-attention layers to maintain the model's core expressive capability.
Behind this architectural convergence is an underestimated driver — NVIDIA Research. Gated DeltaNet was originally proposed by the NVIDIA team at ICLR 2025, essentially introducing a gating mechanism on top of the Mamba2 state-space model, thereby more efficiently using limited state memory. Today, top teams including Qwen and Kimi are doing R&D based on this architecture, and NVIDIA effectively holds the underlying definition rights for this paradigm shift.
For the industry, the hybrid-attention ratio becoming standard means the cost structure of long-context inference will undergo fundamental change. The engineering feasibility of million-token context is greatly improved; the combination of distributed solutions like Ring Attention with hybrid attention may bring ultra-long context from tech showcase to industrial-grade application. The real next question is: while maintaining efficiency advantages, can this architecture further close the gap with pure full-attention models on complex reasoning tasks.