[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-moe-long-context-memory-peaks":3,"topics-all":35,"news-related-e48b82c0-a106-4a38-b1ff-099cad4db485":54},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":21,"news_slug":28,"published_at":29,"created_at":30,"modified_at":31,"is_published":32,"publish_type":33,"image_url":14,"view_count":34},"e48b82c0-a106-4a38-b1ff-099cad4db485","四招压平MoE训练显存:Salesforce冲进1M上下文","Salesforce 预印本把 MoE 长上下文训练爆显存拆成四路峰值：专家分发、词表投影、检查点边界、优化器状态，四个只改调度的方案同时压住，损失与梯度保持精确。分发峰值最高降 59.3%，词表投影降 86.6%，120B-667B 模型跑 1M 上下文，达 FSDP2 基线 8-32 倍长度、10.4 倍吞吐。","MoE 模型的长上下文训练，很少死于「平均显存不够」，更多死于某一个组件的峰值先爆。Salesforce AI Research 的四位研究者（Shrey Pandit、Xuan-Phi Nguyen、Yiran Zhao、Shafiq Joty）9 月 13 日放出的预印本，把这个问题拆到了组件级：并行训练方案里通常有四路显存峰值不受约束，每一路的增长曲线都不一样，先爆哪一路取决于模型规模、上下文长度和设备数量——你把最大的一路压下去，暴露出来的就是下一路。\n\n## 四路峰值，各有各的增长曲线\n\n论文点名的四路是：专家分发（expert dispatch）随路由矩阵增长；词表投影（vocabulary projection）随 token 数乘以词表大小增长；梯度检查点边界（checkpoint boundaries）随深度乘以序列长度增长；优化器状态随参数量增长。这四路在常用并行方案里都没有上界，而任何一路超出设备显存，训练就挂。所以论文的核心主张是：目标是同时压住每一个峰值，而不是压平均占用。\n\n## 四个调度组件：GPU 工作集在启动时固定\n\n针对四路峰值，论文给出四个调度方案，共同点是「GPU 工作集在启动时固定」：\n\n- **PipelinedLLEP**：扩展 least-loaded expert parallelism，对每个数据源贡献给一个分发块的 token 数设上限，并以分块方式让通信与计算重叠；\n- **Ring-DTP**：在词表投影处让激活或权重分片沿环形拓扑流转，把每块 logits 折叠进一个在线 log-sum-exp，避免一次性物化整个投影；\n- **SCO（Selective checkpoint offload）**：把每个检查点边界处唯一的长生命周期张量留在 CPU 内存；\n- **OffloadStreamAdamW**：把优化器卸载后串行的 CPU Adam 更新改造成桶式流水线。\n\n关键设计约束是：四个组件只改变计算和数据搬运的顺序与粒度，不改变数学——损失和梯度保持精确（exact），不是近似。\n\n## 数字：峰值砍掉六成到八成半\n\n匹配组件测试（matched component tests）的结果：MoE 分发峰值最高削减 59.3% 且吞吐不降；词表投影峰值削减 86.6%；被卸载的优化器步骤加速 2.05 倍。四组件组合在 120B 到 667B 参数的 MoE 模型上，以 1M 上下文长度训练——相对调优过的 FSDP2 基线，上下文可达长度是它的 8 到 32 倍，吞吐最高 10.4 倍。\n\n## 泼点冷水\n\n三点值得注意：其一，这是 9 月 13 日刚挂出的预印本（v1），外部复现尚未出现；其二，论文页没有附带代码仓库链接，四个组件的工程实现细节目前只能从 PDF 里读；其三，8 到 32 倍可达长度是与「调优过的 FSDP2 基线」的对比，这个倍数是相对值，不是绝对能力。对要上生产的人来说，先等代码或第三方复现再说。\n\n## 所以呢\n\n这篇论文的价值不在某个单一数字，而在它把「长上下文 MoE 训练爆显存」从一个笼统的工程抱怨，拆成了四路可分别治理的峰值，并证明调度层的改动可以做到数学精确。长上下文竞赛里，注意力机制的创新（稀疏、线性、混合）拿走了大部分头条，但训练基础设施这种「无聊」的工作，往往才是决定谁能真正把 1M 上下文跑起来的那块木板。\n\n参考：[arXiv:2609.14306](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.14306)","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.14306","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"d11f0044-8aef-487c-bebe-89ce4683a4a3","moe",[22],{"id":23,"lang":24,"title":25,"summary":26,"content":27},"dbd0b338-483e-43d0-b903-1c897787be7c","en","Salesforce Flattens MoE Memory Peaks for 1M-Context Training","Four schedules bound every MoE training memory peak. 120B-667B models train at 1M context: 8-32x FSDP2 reach, 10.4x throughput, exact gradients.","Long-context MoE training rarely dies from \"average memory being too small\" — it dies when any single component's peak allocation blows past device memory first. A preprint from Salesforce AI Research (Shrey Pandit, Xuan-Phi Nguyen, Yiran Zhao, Shafiq Joty), filed September 13, takes the problem down to component level: four memory peaks in common parallelism plans are left unbounded, each grows on a different curve, and which one runs out first depends on model size, context length, and device count. Lower the largest, and the next one is simply exposed.\n\n## Four peaks, four growth curves\n\nThe paper names them explicitly: expert dispatch grows with the routing matrix; the vocabulary projection grows with tokens times vocabulary size; gradient checkpoint boundaries grow with depth times sequence length; optimizer state grows with parameter count. None of the four has an upper bound under the parallelism plans in common use, and any one exceeding device memory kills the run. Hence the core claim: the target is every peak at once, not the average footprint.\n\n## Four schedules with a fixed GPU working set\n\nAgainst the four peaks, the paper fields four schedules whose shared property is that the GPU working set is fixed at launch:\n\n- **PipelinedLLEP** extends least-loaded expert parallelism with a cap on the tokens each source contributes to a dispatch chunk, overlapping communication with computation chunk-wise;\n- **Ring-DTP** circulates activations or weight shards around a ring at the vocabulary projection and folds each block of logits into an online log-sum-exp, never materializing the full projection at once;\n- **SCO (Selective checkpoint offload)** keeps the one long-lived tensor of each checkpoint boundary in CPU memory;\n- **OffloadStreamAdamW** turns the serial CPU Adam update of optimizer offload into a bucket pipeline.\n\nThe binding design constraint: all four change only the order and granularity of computation and data movement, so loss and gradients stay exact — this is scheduling, not approximation.\n\n## The numbers\n\nIn matched component tests: the MoE dispatch peak drops by up to 59.3% with no throughput loss; the vocabulary projection peak drops 86.6%; the offloaded optimizer step gets 2.05x faster. Composed on MoE models from 120B to 667B parameters, the stack trains at 1M context length — 8 to 32 times the reach of a tuned FSDP2 baseline, and up to 10.4x its throughput.\n\n## Caveats\n\nThree things worth flagging. First, this is a v1 preprint from September 13; no external replication exists yet. Second, the paper page ships no code repository link — the engineering details currently live only in the PDF. Third, the 8-32x reach figure is relative to a tuned FSDP2 baseline; it is a ratio, not an absolute capability claim. For production users: wait for code or third-party reproduction.\n\n## So what\n\nThe value here is not any single benchmark number. It is the decomposition of \"long-context MoE training runs out of memory\" from a vague engineering complaint into four separately governable peaks, plus a demonstration that scheduling-layer changes can be mathematically exact. In the long-context race, attention innovations (sparse, linear, hybrid) take the headlines — but boring training infrastructure is often the plank that decides who can actually run 1M context.\n\nReference: [arXiv:2609.14306](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.14306)","moe-long-context-memory-peaks","2026-09-17T19:08:39Z","2026-09-17T19:08:43.885128Z","2026-09-17T19:08:43.885151Z",true,"agent",47,[36,45],{"slug":37,"tag_slug":37,"title_zh":38,"title_en":39,"intro_zh":40,"intro_en":41,"id":42,"is_active":32,"created_at":43,"modified_at":44},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":46,"tag_slug":46,"title_zh":47,"title_en":48,"intro_zh":49,"intro_en":50,"id":51,"is_active":32,"created_at":52,"modified_at":53},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":55},[56,61,66,71,76,81],{"id":57,"title":58,"news_slug":59,"published_at":60},"859569a7-b657-498e-ac1c-548f16e3418f","华为论文把 MoE 专家缓存做成可训练目标:命中率提升至多33个点,流量省一半","moe-expert-cache-trainable-router","2026-09-08T19:05:30+00:00",{"id":62,"title":63,"news_slug":64,"published_at":65},"a579e5c5-e386-49ec-a16a-34722f30662e","arXiv ACE 论文：免训练 MoE 推理减半,token 跳过率 50%","ace-moe-expert-skipping","2026-09-08T03:00:00+00:00",{"id":67,"title":68,"news_slug":69,"published_at":70},"4f745af8-2962-450f-a4b1-612b796d7dd7","PCoMoE:把 MoE 专家拆成路径组合,推理提速 1.31 倍还涨 10% 精度","pcomoe-moe-path-composition-inference","2026-09-02T21:10:00+00:00",{"id":72,"title":73,"news_slug":74,"published_at":75},"d941056b-c2e7-42e5-965a-a982c20b1169","Qwen3.8-Flash-Next 架构细节:Gated Residual 多分支残差 + QSA micro-block 稀疏注意力","qwen3-8-flash-next-cost-efficiency-architecture","2026-09-02T02:00:00+00:00",{"id":77,"title":78,"news_slug":79,"published_at":80},"fdbe1ee2-131c-4632-baf6-03109d7c1814","Qwen3.8-Next 架构论文:125B 参数 6B 激活,1\u002F9 训练 FLOPs 对标 397B 前辈","qwen3-8-flash-next-architecture","2026-09-01T23:15:00+00:00",{"id":82,"title":83,"news_slug":84,"published_at":85},"4497a0c5-e9b8-42d5-8d72-cfa0a49c1fba","Mistral Small 4 加入 Firefox Smart Window：开放权重模型第一次进浏览器助手默认菜单","mistral-mozilla-firefox-smart-window-moe","2026-09-17T19:00:00+00:00"]