[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-sebastian-raschka-llm-arch-gallery-40":3,"topics-all":36,"news-related-c8ba186e-e7a6-40e1-9485-41aeb4de388e":55},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"c8ba186e-e7a6-40e1-9485-41aeb4de388e","Sebastian Raschka 发布 LLM 架构图谱：40+ 开源模型一站式横向对比","5月19日，AI研究者Sebastian Raschka正式上线了LLM Architecture Gallery，将2024年初至2026年春季发布的40余款主流开源大模型的架构图谱汇聚成一站式参考库。\n\n这份图谱的独特之处在于「一图看懂」——每款模型配有高清架构图、紧凑事实表（参数量、上下文窗口、注意力机制、解码器类型等），并附上技术论文和开源实现链接。特别实用的设计是支持按参数量、注意力类型、许可证等维度排序，方便开发者快速横向对比。\n\n从图谱可见几个清晰趋势。首先，MoE（混合专家）已成旗舰模型主流架构：DeepSeek V3（671B总参数，37B激活）、Kimi K2（1T总参数，32B激活）、Qwen3-235B均采用稀疏MoE设计，靠极低的KV cache占用换取大参数量的实际可运行性。其次，GQA（分组查询注意力）和QK-Norm正成为标准配置，即使是Dense模型也在跟进。第三，局部+全局注意力的3:1混合模式（滑动窗口处理局部信息，全局注意力捕获长程依赖）在2026年成为新范式，Gemma 3、Mistral Small 3.1、Qwen3系列均采用这一配置。\n\n对推理优化而言，图谱中提供的KV cache占用数据最有参考价值。同样是 bf16 精度，Llama 3.2 1B每token仅需32 KiB，而DeepSeek V3因MoE稀疏性反而只有68.6 KiB，远低于Mistral Small 3.1的160 KiB。这对部署时的显存规划至关重要。\n\n值得注意的是，开源模型正在收敛。Qwen3、DeepSeek V3、Kimi K2在架构上越来越趋同——MLA\u002FGQA + MoE + MTP（多词元预测）似乎正在成为「标准配方」。这种收敛降低了学习成本，但也意味着差异化正在从架构层转向训练数据和特定任务的微调能力。","https:\u002F\u002Fsebastianraschka.com\u002Fllm-architecture-gallery\u002F","8c758013-1efc-4f1d-bc10-8860362115e7",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":18,"name":19,"slug":19,"description":13,"color":13},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",{"id":21,"name":22,"slug":22,"description":13,"color":13},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"d63d00b6-3bf8-4a85-9c9e-1455b13ef24a","en","Raschka's LLM architecture atlas: 40+ open models compared","On May 19, AI researcher Sebastian Raschka officially launched the LLM Architecture Gallery, gathering architecture diagrams of 40+ mainstream open-source large models released from early 2024 through spring 2026 into a one-stop reference library.\n\nWhat makes this gallery unique is its \"one-glance understanding\" — each model comes with a high-definition architecture diagram, a compact fact sheet (parameter count, context window, attention mechanism, decoder type, etc.), and links to the technical paper and open-source implementation. A particularly practical design is the ability to sort by parameter count, attention type, license, etc., letting developers quickly make lateral comparisons.\n\nA few clear trends emerge from the gallery. First, MoE (Mixture of Experts) has become the mainstream architecture for flagship models: DeepSeek V3 (671B total, 37B activated), Kimi K2 (1T total, 32B activated), Qwen3-235B all use sparse-MoE design, trading extremely low KV cache occupancy for practical runnability at large parameter counts. Second, GQA (Grouped Query Attention) and QK-Norm are becoming standard configurations, even Dense models are following. Third, the 3:1 hybrid of local + global attention (sliding window handles local information, global attention captures long-range dependencies) has become the new paradigm in 2026, used by Gemma 3, Mistral Small 3.1, and the Qwen3 family.\n\nFor inference optimization, the KV cache occupancy data in the gallery is most valuable. At the same bf16 precision, Llama 3.2 1B needs only 32 KiB per token, while DeepSeek V3 — due to MoE sparsity — only needs 68.6 KiB, far below Mistral Small 3.1's 160 KiB. This is critical for deployment-time memory planning.\n\nNotably, open-source models are converging. Qwen3, DeepSeek V3, and Kimi K2 are increasingly similar architecturally — MLA\u002FGQA + MoE + MTP (Multi-Token Prediction) seems to be becoming a \"standard recipe.\" This convergence lowers the learning cost, but also means differentiation is shifting from the architecture layer to training data and task-specific fine-tuning capability.","sebastian-raschka-llm-arch-gallery-40","2026-05-19T07:01:00Z","2026-05-19T07:10:04.613082Z","2026-08-19T02:08:40.142862Z",true,"agent",150,[37,46],{"slug":38,"tag_slug":38,"title_zh":39,"title_en":40,"intro_zh":41,"intro_en":42,"id":43,"is_active":33,"created_at":44,"modified_at":45},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":47,"tag_slug":47,"title_zh":48,"title_en":49,"intro_zh":50,"intro_en":51,"id":52,"is_active":33,"created_at":53,"modified_at":54},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":56},[57,62,67,72,77,82],{"id":58,"title":59,"news_slug":60,"published_at":61},"926f89fc-5ed5-4170-bfac-931d3a31b6a4","腾讯混元开源 AngelSpec 投机解码框架：DFly 在 Hy3-A21B 上取得 1.98–2.40× 加速","tencent-angelspec-spec-decoding-hy3-dfly","2026-07-30T00:00:00+00:00",{"id":63,"title":64,"news_slug":65,"published_at":66},"28c41f06-d20f-481c-b133-cd109af3aed1","答对之后停不下来:微软团队揪出在线蒸馏的 EOS 错配元凶","eos-mismatch-opd-length-inflation","2026-09-18T21:09:06+00:00",{"id":68,"title":69,"news_slug":70,"published_at":71},"2e27016d-b90e-45c7-825a-41fd1e435c80","JHU 新研究:组合持续学习机制,百任务记忆留存从 1.2% 提到 34.9%","compose-cl-long-horizon-memorization","2026-09-16T15:10:00+00:00",{"id":73,"title":74,"news_slug":75,"published_at":76},"30fca629-bace-4832-9789-b44aa8c8989d","学生团队从零训出开源 7B 模型 ZGCM-1:数学推理硬刚 235B 前沿","zgcm-1-open-7b-foundation-model","2026-09-15T19:10:00+00:00",{"id":78,"title":79,"news_slug":80,"published_at":81},"2266cea6-06f1-4932-8905-1bc3f2e5a8c0","Meta FAIR 字节蒸馏研究:End-Of-Token 渐近反超 token 蒸馏 4%,数据只需 1\u002F6","meta-fair-byte-distillation-token-ceiling-2026-09","2026-09-15T02:00:00+00:00",{"id":83,"title":84,"news_slug":85,"published_at":86},"2731ed1c-17c3-4d85-9174-983cf50743e3","地铁售票机上的 AI 大考:2.6GB 端侧模型 91.32 分超 GPT-5.6,规则基线也拿 84.6","metrollm-bench-transit-kiosk-llm","2026-09-12T23:08:18+00:00"]