[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-qwen-3-5-shared-expert-hybrid-attention-amd":3,"topics-all":36,"news-related-88298830-f6d5-4a61-ad94-27daaa568a9a":55},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"88298830-f6d5-4a61-ad94-27daaa568a9a","Qwen 3.5 新一代 MoE 架构解读：Shared Expert + 混合注意力重塑长上下文效率","阿里巴巴于2026年2月发布的Qwen 3.5系列，在标准MoE（混合专家）架构基础上引入了两项关键技术改进——Shared Expert共享专家机制与Hybrid Attention混合注意力架构。这两项创新从训练稳定性和推理效率两个维度重新定义了开源大模型的工程可行性。\n\n在传统MoE架构中，每个Token仅被路由到Top-K个专家子网络，其余专家闲置。这种设计虽能降低激活参数和推理成本，但容易出现专家负载不均、训练不稳定等问题。Qwen 3.5引入了Shared Expert机制：额外设置一条专用Dense MLP路径，每个Token都必须经过它提取跨领域通用特征（如语法结构、语义实体、世界知识）。这些通用表征被注入所有路由专家的输入，形成先打底、再专精的计算流程。\n\nAMD在MI300X GPU上的实测验证了效果：相同硬件条件下，Qwen 3.5-397B-A17B的训练收敛速度比Qwen 3提升约18%，下游任务平均准确率高出4-6个百分点。\n\nQwen 3.5采用Hybrid Attention策略：每隔4层保留一组标准全注意力层，确保对高关联性Token的精确召回；中间层替换为Gated Delta Networks，实现对序列长度的线性复杂度扩展。AMD实测数据显示，在超过32K Token的长上下文场景中，Qwen 3.5的吞吐量是Qwen 3的3.2倍，首Token延迟降低57%。\n\n这一架构改进指向更宏大的趋势：2026年开源大模型正从\"暴力Scaling\"向\"架构效率优先\"转型。DeepSeek V4用Muon优化器+FP4压缩训练成本，Qwen 3.5用Shared Expert+线性注意力压缩推理成本，最终目标都是让100B以上参数的开源模型在合理成本下真正可部署、可实用。","https:\u002F\u002Fwww.amd.com\u002Fen\u002Fdeveloper\u002Fresources\u002Ftechnical-articles\u002F2026\u002Fday-0-support-for-qwen-3-5-on-amd-instinct-gpus.html","09817576-1b8d-491e-b843-2913b7bcbe49",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"e9df3399-8d46-4f25-bae9-15c59c532473","en","Qwen 3.5 MoE: shared experts and hybrid attention explained","The Qwen 3.5 series, released by Alibaba in February 2026, introduces two key technical improvements on top of standard MoE (Mixture of Experts) architecture — the Shared Expert mechanism and the Hybrid Attention architecture. These two innovations redefine the engineering feasibility of open-source LLMs from the two dimensions of training stability and inference efficiency.\n\nIn traditional MoE architecture, each Token is only routed to Top-K expert subnetworks, with other experts idle. While this design can reduce active parameters and inference cost, it easily leads to issues like expert load imbalance and training instability. Qwen 3.5 introduces the Shared Expert mechanism: an additional dedicated Dense MLP path is set up, through which every Token must pass to extract cross-domain general features (such as syntactic structure, semantic entities, world knowledge). These general representations are injected into the input of all routing experts, forming a \"lay foundation first, then specialize\" computation flow.\n\nAMD's real-world testing on MI300X GPUs verified the effect: under the same hardware conditions, Qwen 3.5-397B-A17B's training convergence speed is about 18% faster than Qwen 3, with average accuracy 4-6 percentage points higher on downstream tasks.\n\nQwen 3.5 adopts the Hybrid Attention strategy: a group of standard full-attention layers is kept every 4 layers, ensuring precise recall of highly correlated Tokens; the middle layers are replaced with Gated Delta Networks, achieving linear-complexity scaling for sequence length. AMD's measured data shows that in long-context scenarios beyond 32K Tokens, Qwen 3.5's throughput is 3.2× that of Qwen 3, with first-token latency reduced by 57%.\n\nThis architectural improvement points to a broader trend: 2026's open-source LLMs are transitioning from \"brute-force Scaling\" to \"architecture-efficiency first.\" DeepSeek V4 uses Muon optimizer + FP4 to compress training cost, Qwen 3.5 uses Shared Expert + linear attention to compress inference cost — both aimed at making 100B+ parameter open-source models truly deployable and usable at reasonable cost.","qwen-3-5-shared-expert-hybrid-attention-amd","2026-05-05T07:05:00Z","2026-05-05T07:08:26.859852Z","2026-08-19T02:08:40.142862Z",true,"agent",488,[37,46],{"slug":38,"tag_slug":38,"title_zh":39,"title_en":40,"intro_zh":41,"intro_en":42,"id":43,"is_active":33,"created_at":44,"modified_at":45},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":47,"tag_slug":47,"title_zh":48,"title_en":49,"intro_zh":50,"intro_en":51,"id":52,"is_active":33,"created_at":53,"modified_at":54},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":56},[57,62,67,72,77,82],{"id":58,"title":59,"news_slug":60,"published_at":61},"530e5aa5-2026-4c30-b4e6-421caca907b2","Transformer 提前罢工:13 个基座模型跟不住引用链,一个 rank-8 LoRA 修好","tiny-lora-frozen-transformer-chain-relay","2026-10-03T21:05:16+00:00",{"id":63,"title":64,"news_slug":65,"published_at":66},"98312e5c-0687-41f5-a5b3-596a45563e67","DeepSeek V4训练内幕：Muon优化器与FP4混合精度重塑大模型效率","deepseek-v4-muon-optimizer-fp4-mixed-precision","2026-04-28T01:01:00+00:00",{"id":68,"title":69,"news_slug":70,"published_at":71},"3a419e1c-fb2b-467e-a5b6-a7188525bc03","TraceForge:推理痕迹自己造,学生反超教师真迹","traceforge-family-native-distillation","2026-10-22T15:10:42+00:00",{"id":73,"title":74,"news_slug":75,"published_at":76},"fd29c2f1-175f-416e-bde2-95dd65ca3465","KronLSTM 精准改写记忆:线性注意力长上下文新解","kronlstm-kronecker-delta-rule","2026-10-20T15:06:16+00:00",{"id":78,"title":79,"news_slug":80,"published_at":81},"559df680-f13e-48f0-9263-73116aaff411","上交大DLite免历史投机解码:Agent提速2.74倍","dlite-history-free-speculative-decoding","2026-10-19T17:07:52+00:00",{"id":83,"title":84,"news_slug":85,"published_at":86},"ab1e6bdf-9433-41f0-be4d-cc0f78523ac2","SonicBoom v0.0.3:一个 .so 干掉手工调优","sonicboom-v003-mlir-compiler","2026-10-19T09:03:05+00:00"]