In April-May 2026, open-source LLMs welcomed the most dense new-release period in history. Meta Llama 4, Alibaba Qwen 3.6, Google Gemma 4, DeepSeek V4, Mistral Medium 3.5, and Moonshot Kimi K2.6 have taken the stage one after another. Behind this release wave, a clear technical-path divergence is emerging: MoE (Mixture of Experts) architecture is becoming the mainstream.
Architecturally, both Llama 4 Scout and Maverick use 17B active parameters MoE design; Scout activates only 16 experts within 109B total parameters, while Maverick extends to 128 experts with 400B total. Qwen 3.6-235B's MoE config activates about 22B parameters, and DeepSeek V4 Pro drives 1.6T total scale with 49B active parameters. The three highly consistent in their choice: use sparse activation to trade for exponential parameter scaling while keeping inference cost controllable.
In contrast, Google's Gemma 4 and Mistral's Medium 3.5 chose Dense architecture. Gemma 4-31B uses 31B dense parameter design, while Mistral Medium 3.5 is a 128B pure dense model, neither using MoE sparse activation. These two choices represent different engineering philosophies: Dense architecture has stronger consistent-output capability on specific tasks, but for a given active-parameter budget, the total knowledge capacity accessible is limited by the parameter count.
Benchmark data confirms this divergence. DeepSeek V4 Pro hits 80.6% on SWE-Bench Verified, Kimi K2.6 at 80.2%, both MoE architecture. Mistral Medium 3.5 follows at 77.6%, but dense architecture's knowledge capacity at the same active scale is far below MoE models — sparse activation lets the same active parameters encode more specialized knowledge.
The current open-source ecosystem has entered a refined-development stage. The MoE camp is represented by DeepSeek V4, Kimi K2.6, Qwen 3.6, while the Dense camp is led by Gemma 4 and Mistral Medium 3.5. The technical-path divergence leaves developers facing a real choice: sparse activation for scale advantages, or dense architecture for output stability? The answer to this question will depend on the inference budget and task characteristics of specific application scenarios.