The Qwen 3.5 series, released by Alibaba in February 2026, introduces two key technical improvements on top of standard MoE (Mixture of Experts) architecture — the Shared Expert mechanism and the Hybrid Attention architecture. These two innovations redefine the engineering feasibility of open-source LLMs from the two dimensions of training stability and inference efficiency.
In traditional MoE architecture, each Token is only routed to Top-K expert subnetworks, with other experts idle. While this design can reduce active parameters and inference cost, it easily leads to issues like expert load imbalance and training instability. Qwen 3.5 introduces the Shared Expert mechanism: an additional dedicated Dense MLP path is set up, through which every Token must pass to extract cross-domain general features (such as syntactic structure, semantic entities, world knowledge). These general representations are injected into the input of all routing experts, forming a "lay foundation first, then specialize" computation flow.
AMD's real-world testing on MI300X GPUs verified the effect: under the same hardware conditions, Qwen 3.5-397B-A17B's training convergence speed is about 18% faster than Qwen 3, with average accuracy 4-6 percentage points higher on downstream tasks.
Qwen 3.5 adopts the Hybrid Attention strategy: a group of standard full-attention layers is kept every 4 layers, ensuring precise recall of highly correlated Tokens; the middle layers are replaced with Gated Delta Networks, achieving linear-complexity scaling for sequence length. AMD's measured data shows that in long-context scenarios beyond 32K Tokens, Qwen 3.5's throughput is 3.2× that of Qwen 3, with first-token latency reduced by 57%.
This architectural improvement points to a broader trend: 2026's open-source LLMs are transitioning from "brute-force Scaling" to "architecture-efficiency first." DeepSeek V4 uses Muon optimizer + FP4 to compress training cost, Qwen 3.5 uses Shared Expert + linear attention to compress inference cost — both aimed at making 100B+ parameter open-source models truly deployable and usable at reasonable cost.