[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-qwen3-8-flash-next-cost-efficiency-architecture":3,"news-related-d941056b-c2e7-42e5-965a-a982c20b1169":45},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":30,"news_slug":37,"published_at":38,"created_at":39,"modified_at":40,"is_published":41,"publish_type":42,"image_url":43,"view_count":44},"d941056b-c2e7-42e5-965a-a982c20b1169","Qwen3.8-Flash-Next 架构细节:Gated Residual 多分支残差 + QSA micro-block 稀疏注意力","Qwen 发布 Qwen3.8-Flash-Next 架构论文:125B 总参数 + 6B 激活的稀疏 MoE,搭配 Gated DeltaNet、QSA 稀疏注意力、Gated Residual 多分支残差与 Muon,以约 1\u002F9 训练算力在 14 项预训练任务中 8 项反超自家 397B-A17B 前辈。","## Qwen3.8-Flash-Next 把 125B MoE 跑赢 397B 的前辈,但 1\u002F9 训练算力\n\n阿里通义千问团队本周公开了 Qwen3.8-Flash-Next 的完整架构报告和技术博客,核心结论相当扎眼:一个 125B 总参数、6B 激活参数的稀疏 MoE,只用约 1\u002F9 的训练 FLOPs,在 14 个预训练 benchmark 上 8 项反超自家 397B-A17B 的前辈,其余 6 项差距不超过 2.6 分。这不只是又一份「更大更强」式发布,真正的看点是它一次性把 Qwen 系列未来要走的几条架构路线 —— 注意力、残差、嵌入、优化器 —— 都打包进了同一个开源权重。来源:[Qwen 官方博客, 2026-08-26](https:\u002F\u002Fqwen.ai\u002Fblog?id=qwen3.8-flash-next),以及 [arXiv:2608.30320, 2026-08-31](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.30320)。\n\n## 四个层面的同时升级\n\nQwen 团队把这次架构迭代拆成四块讲:\n\n- **注意力层**:沿用并强化「Gated DeltaNet(GDN)+ 全注意力」的层间混合模式,平均每 4 层放 1 层全注意力;继续预训练阶段再把全注意力层替换为新设计的 Qwen Sparse Attention(QSA),用一个轻量索引器在 micro-block 粒度上打分。\n- **残差流**:从单条分支扩到 4 条并行分支,用一个 elementwise gate 把它们汇合回主干,这一设计被他们命名为 Gated Residual(GR)。\n- **嵌入层**:在骨干之外单独加了一层 n-gram embedding,参数表约 51B,放在 host memory 预取,不占加速器显存 —— 等于「白送」一层容量。\n- **优化器**:继续用 Muon,带来更优的学习率与 batch size 区间,论文里说「不再需要 batch-size warmup,在压力测试下稳定性大幅改善」。\n\n## 怎么用 1\u002F9 算力追平 397B\n\n数字最直观的对照是这一句:125B 总参数 + 6B 激活 + 1\u002F3 训练 token + 约 1\u002F9 训练 FLOPs,在 14 个预训练任务里 8 项领先、6 项小幅落后。论文特别强调这是「loss 与下游准确率不再同步变化」的典型案例 —— 单调扩大 n-gram 词表能持续压低 loss,但下游指标很快就饱和,反而是 GR、QSA、Muon 三件套共同把帕累托前沿往左上推。\n\n另外值得注意的是,模型在更短的训练 token 上达到了接近的水平,这意味着 QSA 替换后的长上下文推理、以及 GDN 在短上下文的高效推理,共同把每 token 的训练成本打到了更低的水位 —— 这一项对开源社区的复现成本影响最大,而不是单纯一个 benchmark 跑分。\n\n## 为什么这是 Qwen4 的「早期预览」\n\n官方博客里有一句很直白的定位:Qwen3.8-Flash-Next 是「Qwen4 架构的早期预览」,延续了 Qwen3-Next 当年在 Qwen3.5 之前提前放架构的惯例。换句话说,Qwen 团队把这一波架构创新拆成两步:先用权重开源的方式让社区验证,再在完整 Qwen4 上把这些设计堆到更大规模。这是一条和「先放大参数、再扩训练 token」的常规 scaling path 不同的路线 —— 它优先押注「架构效率」而非「参数膨胀」。\n\n这条路线和 2026 年开源 LLM 的几条主旋律是吻合的:GLM-5.3-Flash 用 320B 总参数 \u002F 18B 激活打对标 Opus 的 Flash 价位,Qwen3.8-Max 把 2.4T 的超大模型也走开源路线但改用收入分成许可证,Kimi K3、DeepSeek V4 都在用 MoE 稀疏化去拉平推理成本。当「Scaling Law 边际收益递减 + 推理算力吃紧」成为共识,把每一瓦时、每一张卡用得更聪明,正在取代「再训一个更大的稠密模型」成为新的竞争维度。\n\n## 所以呢\n\n如果你在做开源 LLM 训练或推理优化,这份报告值得花时间精读,尤其是 QSA 的 micro-block 索引器、GR 的多分支残差、以及 n-gram embedding 的 host-prefetch 设计 —— 三者都是可以独立移植的模块化改动,不需要重训整套 backbone。如果你只是消费者,Qwen3.8-Flash-Next-Base 已经可以直接在 Hugging Face 拉权重,6B 激活意味着单张 H100 就能跑推理,加上 vLLM、FlashQLA 等推理框架已经合并相关算子,本地落地成本比上一代旗舰低一个数量级。当「参数更大」不再是新闻,把 1\u002F9 算力追平前辈这件事,本身就是下半年开源 LLM 真正的看点。原文链接:[Qwen3.8-Flash-Next 官方博客](https:\u002F\u002Fqwen.ai\u002Fblog?id=qwen3.8-flash-next),[arXiv 技术报告](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.30320),[vLLM Recipes 部署指南](https:\u002F\u002Frecipes.vllm.ai\u002FQwen\u002FQwen3.8-Flash-Next)。","https:\u002F\u002Fqwen.ai\u002Fblog?id=qwen3.8-flash-next","c36a21ac-2a77-421b-9519-1e150695732a",[11,15,18,21,24,27],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"7e89b5cc-57db-4f37-bc6d-28919a73931c","model-release",{"id":22,"name":23,"slug":23,"description":14,"color":14},"d11f0044-8aef-487c-bebe-89ce4683a4a3","moe",{"id":25,"name":26,"slug":26,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",{"id":28,"name":29,"slug":29,"description":14,"color":14},"c187600e-804c-4697-b828-1e4330e0eb10","qwen",[31],{"id":32,"lang":33,"title":34,"summary":35,"content":36},"ef6ee594-facd-4e6f-b183-81d267d1d03d","en","Qwen3.8-Flash-Next architecture: 125B parameters at 6B activated, matches 397B predecessor at 1\u002F9 training FLOPs","The Qwen team published the Qwen3.8-Flash-Next architecture paper and tech blog: a 125B-total \u002F 6B-activated sparse MoE with Gated DeltaNet, QSA sparse attention, Gated Residual multi-branch residuals and Muon, matches or beats the 397B-A17B predecessor on 8 of 14 pre-training benchmarks at roughly 1\u002F9 the training compute.","## Qwen3.8-Flash-Next matches a 397B MoE with 125B parameters at 6B activated, on 1\u002F9 the training compute\n\nAlibaba's Qwen team this week released the full architecture report and tech blog for Qwen3.8-Flash-Next, and the headline number is striking: a sparse MoE with 125B total parameters and only 6B activated per token, trained on roughly 1\u002F9 the FLOPs of its predecessor, still beats that 397B-A17B model on 8 of 14 pre-training benchmarks and trails it by no more than 2.6 points on the rest. This is more than another \"bigger is better\" release — the real story is that Qwen has bundled four architectural directions (attention, residuals, embeddings, optimizer) into a single open-weight drop. Sources: [Qwen official blog, 2026-08-26](https:\u002F\u002Fqwen.ai\u002Fblog?id=qwen3.8-flash-next) and [arXiv:2608.30320, 2026-08-31](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.30320).\n\n## Four layers of the architecture moved at once\n\nThe team breaks the iteration into four components:\n\n- **Attention layer**: extends the hybrid \"Gated DeltaNet (GDN) + full attention\" pattern, with one full-attention layer for every four GDN layers. During continued pre-training those full-attention layers are replaced by a new design, Qwen Sparse Attention (QSA), which scores context at micro-block granularity with a lightweight indexer.\n- **Residual stream**: widened from a single branch to four parallel branches, recombined through an elementwise gate. The team names this design Gated Residual (GR).\n- **Embedding layer**: a single n-gram embedding table of roughly 51B parameters lives outside the backbone, prefetched from host memory so it does not occupy accelerator HBM — effectively \"free\" capacity.\n- **Optimizer**: continued use of Muon, which shifts the optimal learning rate and batch size upward, eliminates the need for batch-size warmup, and dramatically improves stability under stress tests.\n\n## How 1\u002F9 the compute still catches a 397B\n\nThe most concrete comparison is this: 125B total \u002F 6B activated \u002F 1\u002F3 the training tokens \u002F roughly 1\u002F9 the training FLOPs, leading the predecessor on 8 of 14 pre-training tasks and slightly trailing on the other 6. The paper highlights that loss and downstream accuracy stop moving together here — monotonically enlarging the n-gram vocabulary keeps driving loss down, but downstream scores saturate quickly. It is the combination of GR, QSA and Muon that actually pushes the Pareto frontier up and to the left.\n\nEqually important is what the model achieves with fewer training tokens: QSA at long context and GDN at short context together pull per-token training cost down to a new low. This matters far more for community reproducibility than any single benchmark score.\n\n## Why this is an early preview of Qwen4\n\nThe blog is unusually explicit: Qwen3.8-Flash-Next is \"an early preview of the architecture used in Qwen4\", continuing the pattern set when Qwen3-Next shipped ahead of Qwen3.5. The team is splitting this wave of architectural innovation into two steps — first ship the open weights for community validation, then stack the same design onto the full Qwen4 family. It is a different scaling path from the conventional \"grow parameters, then grow tokens\" recipe; the bet here is on architectural efficiency rather than parameter inflation.\n\nThat bet lines up with the broader 2026 open-weight playbook: GLM-5.3-Flash pairs 320B total \u002F 18B activated with a Flash-tier API price targeting Opus, Qwen3.8-Max open-sources a 2.4T parameter model under a revenue-sharing license, and Kimi K3 \u002F DeepSeek V4 both lean on MoE sparsity to flatten inference cost. As diminishing returns on scaling collide with constrained inference compute, getting more out of every watt and every accelerator is replacing \"train an even bigger dense model\" as the real competitive axis.\n\n## So what\n\nFor anyone working on open-weight LLM training or inference optimization, this paper deserves a close read — especially the micro-block indexer in QSA, the multi-branch residual in GR, and the host-prefetch design of the n-gram embedding. All three are modular changes that can be lifted into other stacks without retraining the whole backbone. For consumers, Qwen3.8-Flash-Next-Base weights are already live on Hugging Face, and with only 6B parameters activated per token, a single H100 is enough for inference; vLLM and FlashQLA have already merged the relevant kernels, so on-device cost is an order of magnitude lower than the last generation of flagships. When \"bigger parameters\" stops being news, \"matching the previous generation on one ninth of the compute\" is exactly the story worth watching through the second half of 2026. Original links: [Qwen3.8-Flash-Next official blog](https:\u002F\u002Fqwen.ai\u002Fblog?id=qwen3.8-flash-next), [arXiv technical report](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.30320), [vLLM Recipes deployment guide](https:\u002F\u002Frecipes.vllm.ai\u002FQwen\u002FQwen3.8-Flash-Next).","qwen3-8-flash-next-cost-efficiency-architecture","2026-09-02T02:00:00Z","2026-09-02T11:10:28.604671Z","2026-09-02T11:10:28.604685Z",true,"agent","https:\u002F\u002Fqianwen-res.oss-accelerate.aliyuncs.com\u002FQwen3.8-Flash-Next\u002FQwen3.8-flash_banner_en.jpg",129,{"items":46},[47,52,57,62,67,72],{"id":48,"title":49,"news_slug":50,"published_at":51},"fdbe1ee2-131c-4632-baf6-03109d7c1814","Qwen3.8-Next 架构论文:125B 参数 6B 激活,1\u002F9 训练 FLOPs 对标 397B 前辈","qwen3-8-flash-next-architecture","2026-09-01T23:15:00+00:00",{"id":53,"title":54,"news_slug":55,"published_at":56},"637f84e0-e6dc-490a-bba1-879f6527bdd5","Qwen3.8-Max 2.4T 开源:Gated DeltaNet 把长上下文成本砍到 1\u002F8","qwen3-8-max-2-4t-open-weights-gated-deltanet","2026-08-30T03:00:00+00:00",{"id":58,"title":59,"news_slug":60,"published_at":61},"33f3b08b-c8a2-43ec-81cf-85e2b918f913","腾讯开源 Hy4 preview:770B MoE、1M 上下文,模型首次参与自身训练","tencent-hy4-preview-770b-moe","2026-08-29T15:00:00+00:00",{"id":63,"title":64,"news_slug":65,"published_at":66},"3d36921f-3b84-4663-97a0-fee7d4eff795","汤森路透开源 Thomson-1.0-Small:持续学习改造 Qwen,3B 激活的 35B MoE","thomson-1-0-small-continual-learning","2026-08-28T19:10:00+00:00",{"id":68,"title":69,"news_slug":70,"published_at":71},"c4027891-42ca-4517-817a-83a48550b1bb","Qwen3.8-Flash-Next 开源:6B 激活参数跑赢 Opus,训练成本仅 1\u002F9","qwen-flash-next-gdn-qsa-architecture","2026-08-27T15:10:00+00:00",{"id":73,"title":74,"news_slug":75,"published_at":76},"804ab59a-a8d6-4b61-bf74-8f6f2bdae83c","智谱把 Flash 做成一件正经事:一次说清 GLM-5.3-Flash 的架构和 benchmark 真相","glm-5-3-flash-hybrid-attention-architecture","2026-08-27T08:00:00+00:00"]