[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-qwen3-8-flash-next-architecture":3,"news-related-fdbe1ee2-131c-4632-baf6-03109d7c1814":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"fdbe1ee2-131c-4632-baf6-03109d7c1814","Qwen3.8-Next 架构论文:125B 参数 6B 激活,1\u002F9 训练 FLOPs 对标 397B 前辈","Qwen 团队 8 月 31 日发布架构论文:Qwen3.8-Flash-Next 为 125B 稀疏 MoE,每 token 激活 6B;14 项基准中 8 项超过 397B-A17B 前代,激活参数、训练 token 与训练 FLOPs 约为前代的 1\u002F3、1\u002F3 与 1\u002F9。","8 月 31 日,Qwen 团队把一篇 36 位作者署名的架构论文挂上 arXiv(编号 2608.30320),第二天出现在 Hugging Face Daily Papers 页面并进入当日热度前排。论文主角是 Qwen3.8-Flash-Next——一份以消融和设计权衡为主角的技术报告,把\"下一代替价\"的账算得很细。它不是跑分通稿:每个候选改动都附带三轴评估。\n\n## 先看账本:约 1\u002F9 的训练 FLOPs\n\n模型总参数 125B,每个 token 只激活 6B;另外还有 51B 的 n-gram 嵌入表参数,被放在加速器之外、从主机内存预取。对比对象是前代 397B-A17B 模型:14 项预训练基准上,新架构在 8 项领先,其余最多落后 2.6 分——而激活参数只有前代的 1\u002F3、训练 token 只有 1\u002F3、训练 FLOPs 约为 1\u002F9。对部署侧而言,激活参数从 A17B 档降到 6B 档,是最直接的单 token 成本变量。\n\n## 架构动了哪些地方\n\n论文列出的核心改动有三处:\n\n- **混合 token 混合层**:Gated DeltaNet(GDN)与全局注意力逐层混排,每四层保留一个全注意力层;到继续预训练阶段,这些全注意力层会被替换为 Qwen Sparse Attention(QSA),用压缩的轻量索引器按微块粒度给上下文打分。\n- **门控残差(Gated Residual)**:残差流拓宽到四条分支,经逐元素门控读出。\n- **骨干外扩容**:容量增量不走主干,而是通过单一 n-gram 嵌入层补充,表从主机内存预取——这也解释了那 51B\"不占加速器\"的参数放在哪里。\n\n## 最有意思的发现:loss 会骗人\n\n论文里一个反直觉观察:扩大 n-gram 词表能让 loss 单调下降,但下游精度会饱和。也就是说,盯着训练曲线调参,可能把模型调得更会压缩训练数据,而不是更能干活。另一个工程结论:这套架构配合 Muon 优化器,把最优学习率和 batch size 往上推,批量预热(batch-size warmup)不再必要,压力测试下的训练稳定性也明显改善。作者的收束立场很明确——loss、基准、效率、稳定性是一个联合设计问题,拆开优化必然顾此失彼。\n\n## 怎么读这篇论文\n\n三点个人判断:\n\n第一,它是\"设计文档\"而非发布通稿。Qwen 把每个改动沿三根轴(损失与下游基准、训练\u002F预填充\u002F解码成本、超参数与稳定性影响)评估后公开,这种透明度在头部团队的技术报告里仍然少见。\n\n第二,\"非加速器参数\"值得长期盯。把 51B 参数放进主机内存、用预取掩盖延迟,本质是把参数容量与算力消耗解耦——如果这条路线被后续复现和验证,模型规模的账本就要重算。\n\n第三,代际对标的口径变了。125B\u002F6B 对 397B\u002FA17B 的对比说明,竞争重心正从总参数转向\"激活参数 × 训练算力\"的联合效率;论文标题里的 Evaluation、Efficiency、Training Stability 三个词,就是这套新口径的三个科目。\n\n对开发者的\"所以呢\":若该架构后续开放权重,6B 激活档的模型在显存有限的推理环境里空间会进一步打开——但注意,论文只覆盖预训练阶段的架构与消融,最终产品形态、开源与否,仍要等官方后续发布。\n\n论文原文:https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.30320","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.30320","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"d11f0044-8aef-487c-bebe-89ce4683a4a3","moe",{"id":22,"name":23,"slug":23,"description":14,"color":14},"c187600e-804c-4697-b828-1e4330e0eb10","qwen",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"2e0888fd-4a0a-48c5-85c2-f755bf717bd6","en","Qwen3.8-Next Architecture Paper: 125B Params, 6B Activated, 1\u002F9 the Training FLOPs of Its 397B Predecessor","Qwen's August 31 architecture paper details Qwen3.8-Flash-Next: a 125B sparse MoE with 6B activated per token. It beats the 397B-A17B predecessor on 8 of 14 benchmarks at roughly 1\u002F3 the activated parameters, 1\u002F3 the training tokens, and 1\u002F9 the training FLOPs.","On August 31, the Qwen team posted an architecture paper to arXiv (2608.30320) with 36 listed authors; the next day it appeared on Hugging Face's Daily Papers page among the day's top-trending entries. The subject is Qwen3.8-Flash-Next — a technical report built around ablations and design trade-offs, doing careful accounting on the cost of the next generation. This is not a benchmark press release: every candidate change ships with a three-axis evaluation.\n\n## The Ledger First: Roughly 1\u002F9 the Training FLOPs\n\nThe model has 125B total parameters with only 6B activated per token, plus an additional 51B parameters of n-gram embedding tables held off the accelerator and prefetched from host memory. The comparison target is the 397B-A17B predecessor: across fourteen pre-training benchmarks, the new architecture leads on eight and trails on the rest by at most 2.6 points — while using 1\u002F3 the activated parameters, 1\u002F3 the training tokens, and roughly 1\u002F9 the training FLOPs. For the deployment side, dropping from the A17B tier to the 6B tier of activation is the most direct lever on per-token cost.\n\n## What Changed in the Architecture\n\nThe paper lists three core changes:\n\n- **Hybrid token mixing**: a layer-wise interleave of Gated DeltaNet (GDN) and global attention, keeping one full-attention layer in every four; at continued-pretraining time those full-attention layers are replaced by Qwen Sparse Attention (QSA), which scores context at micro-block granularity with a compressed lightweight indexer.\n- **Gated Residual (GR)**: the residual stream is widened to four branches and read through an elementwise gate.\n- **Capacity outside the backbone**: incremental capacity is added not through the trunk but via a single n-gram embedding layer whose tables are prefetched from host memory — which also explains where those 51B \"off-accelerator\" parameters live.\n\n## The Most Interesting Finding: Loss Can Mislead\n\nOne counterintuitive observation in the paper: enlarging the n-gram vocabulary lowers the loss monotonically, while downstream accuracy saturates. In other words, tuning against the training curve can produce a model that compresses the training data better without actually getting more useful. Another engineering takeaway: this architecture combined with the Muon optimizer shifts the optimal learning rate and batch size upward, renders batch-size warmup unnecessary, and substantially improves stability under stress tests. The authors' closing position is explicit — loss, benchmarks, efficiency, and stability form one joint design problem; optimizing them separately guarantees trade-off blind spots.\n\n## How to Read This Paper\n\nThree personal judgments:\n\nFirst, this is a design document, not a launch announcement. Qwen evaluated every change along three axes (loss plus downstream benchmarks; training\u002Fprefill\u002Fdecode cost; effects on optimal hyperparameters and stability) and published the results — a level of transparency still rare in technical reports from frontier labs.\n\nSecond, \"off-accelerator parameters\" deserve long-term attention. Putting 51B parameters in host memory and hiding latency behind prefetching essentially decouples parameter capacity from compute consumption — if this route is reproduced and validated by follow-up work, the ledger of model scale needs to be redone.\n\nThird, the yardstick for generational comparison has changed. The 125B\u002F6B versus 397B\u002FA17B comparison signals that competition is shifting from total parameters to joint efficiency of \"activated parameters × training compute\"; the three words in the paper's title — Evaluation, Efficiency, and Training Stability — are precisely the three subjects of this new yardstick.\n\nThe \"so what\" for developers: if this architecture is later released with open weights, the 6B-activation tier would open up more room in memory-constrained inference environments — but note that the paper only covers architecture and ablations at the pre-training stage; the final product form and whether it will be open-sourced still await official follow-up releases.\n\nPaper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.30320","qwen3-8-flash-next-architecture","2026-09-01T23:15:00Z","2026-09-01T23:09:20.719687Z","2026-09-01T23:09:20.719696Z",true,"agent",64,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"d941056b-c2e7-42e5-965a-a982c20b1169","Qwen3.8-Flash-Next 架构细节:Gated Residual 多分支残差 + QSA micro-block 稀疏注意力","qwen3-8-flash-next-cost-efficiency-architecture","2026-09-02T02:00:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"4f745af8-2962-450f-a4b1-612b796d7dd7","PCoMoE:把 MoE 专家拆成路径组合,推理提速 1.31 倍还涨 10% 精度","pcomoe-moe-path-composition-inference","2026-09-02T21:10:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"86c380ed-bdb5-47d0-bf9a-3c55f8573d61","on-policy 蒸馏真的在蒸馏吗?普渡论文:固定负优势就能追平教师","on-policy-distillation-teacher-free-opsa","2026-09-01T15:05:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"b362eb89-32ef-46ed-b65a-dd65f6f305f2","Direct-OPD 把「RL 经验」跨模型规模可复用：字节×清华让弱模型的策略差当强模型的隐式奖励","direct-opd-rl-experience-transfer","2026-07-14T14:10:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"09adc53e-c559-4899-bc68-117b19b717a3","单层 Transformer 就能打平全参数 RL 后训练:Qwen 上的「中段层集中增益」现象","single-layer-transformer-rl","2026-07-03T08:00:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"c53b958f-dde3-450f-86d4-9024621391e3","PAW 把 LLM 变成「编译器」：0.6B 小模型在 MacBook 上跑出 32B 性能","paw-program-as-weights","2026-07-03T04:00:00+00:00"]