In April 2026, Meta released the Llama 4 open-source LLM family, fully transitioning to the Mixture-of-Experts (MoE) hybrid-expert architecture, a change from the previous Dense Transformer route. The Llama 4 series, while maintaining open-source weight availability, approaches or even surpasses closed-source top-tier models like GPT-4o and Gemini 2.0 Flash on multiple benchmarks, seen as the most important architectural upgrade in open-source LLM history.
The Llama 4 family includes two flagship models: Llama 4 Scout and Llama 4 Maverick. Scout has 109B total parameters, activating only 17B per inference (16 experts), supporting up to 10 million tokens of ultra-long context, able to process entire codebases or book-level tasks at once. The flagship Maverick has 400B total parameters, also activating only 17B per inference (128 experts), breaking 1400 on the LMArena benchmark, surpassing GPT-4o and Gemini 2.0 Flash.
The core logic of MoE architecture is sparse activation: not every token goes through all 400B parameters of computation, but is dynamically routed to the most relevant expert subnetwork. A single 8×H100 GPU node can deliver GPT-4-level quality, with inference cost dropping to about one-fifth of closed-source models. Scout after Int4 quantization can even run on a single H100, greatly lowering the local deployment threshold.
Open-source weights mean free download, quantization, and fine-tuning. Since April, major inference platforms including Together AI and Fireworks AI have launched Llama 4 APIs, and Ollama also supports local one-click pull. For teams constrained by budget or data privacy, Llama 4 Maverick offers comparable capability with significantly lower cost — an efficiency revolution in itself.
From a technical evolution perspective, Llama 4 validates the feasibility of MoE on ultra-large-scale open-source models. It's foreseeable that sparse activation will become the mainstream direction for open-source LLMs.