[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-1fe72544-eeb2-47ad-a849-450ef7056065":3},{"id":4,"title":5,"summary":6,"original_url":7,"source_id":8,"tags":9,"published_at":23,"created_at":24,"modified_at":25,"is_published":26,"publish_type":27,"image_url":13,"view_count":28},"1fe72544-eeb2-47ad-a849-450ef7056065","Token-Superposition Training：Nous Research 用「平均嵌入」两阶段预训练把 10B MoE 训练时长砍掉 2.5 倍","Nous Research 发布的 Token-Superposition Training（TST）是一种零侵入的预训练加速方法：在训练的前 r 步（r∈[0.2,0.4]），把 s 个连续 token 的 embedding 取平均作为一个 latent s-token，模型在 L\u002Fs 序列长度上预测下一个 bag 的 s 个目标 token，用多热交叉熵 MCE loss；之后切回标准 next-token 预测。该方法不动模型架构、optimizer、tokenizer、并行策略和数据，纯靠按 FLOPs 折算文本摄入量获得吞吐收益。3B dense 同 FLOPs 下 loss 由 2.808 降到 2.676，10B-A1B MoE 同 loss 下训练时长由 12,311 降到 4,768 B200-小时（约 2.5x 加速），HellaSwag、ARC-Easy\u002FChallenge、MMLU 全部反超。消融显示输入侧和输出侧是两种正交机制；论文也给出边界条件：仅适合算力受限、数据充足的预训练。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2605.06546","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source","2026-05-13T10:00:00Z","2026-06-05T22:20:24.956772Z","2026-06-05T22:20:24.956782Z",true,"agent",1]