On May 19, AI researcher Sebastian Raschka officially launched the LLM Architecture Gallery, gathering architecture diagrams of 40+ mainstream open-source large models released from early 2024 through spring 2026 into a one-stop reference library.

What makes this gallery unique is its "one-glance understanding" — each model comes with a high-definition architecture diagram, a compact fact sheet (parameter count, context window, attention mechanism, decoder type, etc.), and links to the technical paper and open-source implementation. A particularly practical design is the ability to sort by parameter count, attention type, license, etc., letting developers quickly make lateral comparisons.

A few clear trends emerge from the gallery. First, MoE (Mixture of Experts) has become the mainstream architecture for flagship models: DeepSeek V3 (671B total, 37B activated), Kimi K2 (1T total, 32B activated), Qwen3-235B all use sparse-MoE design, trading extremely low KV cache occupancy for practical runnability at large parameter counts. Second, GQA (Grouped Query Attention) and QK-Norm are becoming standard configurations, even Dense models are following. Third, the 3:1 hybrid of local + global attention (sliding window handles local information, global attention captures long-range dependencies) has become the new paradigm in 2026, used by Gemma 3, Mistral Small 3.1, and the Qwen3 family.

For inference optimization, the KV cache occupancy data in the gallery is most valuable. At the same bf16 precision, Llama 3.2 1B needs only 32 KiB per token, while DeepSeek V3 — due to MoE sparsity — only needs 68.6 KiB, far below Mistral Small 3.1's 160 KiB. This is critical for deployment-time memory planning.

Notably, open-source models are converging. Qwen3, DeepSeek V3, and Kimi K2 are increasingly similar architecturally — MLA/GQA + MoE + MTP (Multi-Token Prediction) seems to be becoming a "standard recipe." This convergence lowers the learning cost, but also means differentiation is shifting from the architecture layer to training data and task-specific fine-tuning capability.