[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-224fc205-6b1a-48d3-8562-df66921d017e":3},{"id":4,"title":5,"summary":6,"original_url":7,"source_id":8,"tags":9,"published_at":23,"created_at":24,"modified_at":25,"is_published":26,"publish_type":27,"image_url":13,"view_count":28},"224fc205-6b1a-48d3-8562-df66921d017e","腾讯把序列长度做成 LLM 第四 scaling 轴：WeLM-HD4-617B 在不增参数前提下反超 Kimi K2.6","腾讯 WeChat AI \u002F WeLM 团队 7 月 9 日发布《Hidden Decoding at Scale》,把序列长度作为新的固定 backbone 扩展轴,通过多流嵌入展开(n=4)+ Stream-Factorized Attention,在 WeLM-80B 和 WeLM-617B 两个 MoE 上做 continued pretraining,9 项基准全胜自身 AR 基线,617B 变体在 GPQA Diamond、PHYBench、MathArena Apex、HMMT、SciCode 上反超 Kimi K2.6,训练成本仅近线性增长(80B 5.1× \u002F 617B 4.4×)。\n\n腾讯 WeChat AI \u002F WeLM 团队 7 月 9 日在 arXiv 放出《Hidden Decoding at Scale》,把「latent computation scaling」思路推到 100B+ MoE 体量。方法不做深度\u002F宽度扩展,而是把每个 token 的 embedding 复制 n 份沿序列方向展开,在同一 Transformer backbone 一次性前向,只有最终流(stream)接 LM head 损失,前面 n-1 流当 latent scratchpad 不受监督——backbone 参数不变,每个 token 在一次前向里获得 n 倍有效计算。\n\n让方法能跑上 100B+ 的关键是 Stream-Factorized Attention:大多数层只做流内因果注意力,少数层做跨流混合,把 n² 增长压到近线性。80B 和 617B 训练单步时间为未扩展基线的 5.1× 和 4.4×,落在「理想 4×」与「全连接 16×」之间。再叠 WeLM backbone 自带的 KV-mirror 设计(后段层 KV 只依赖前段层 hidden state),镜像层只跑最终流,80B 32k 单 batch 再省 20%。\n\n效果上 HD4-80B 与 HD4-617B 在相同 CPT + 早期 SFT-only 协议下 9 项基准相对自身 AR 基线全部正向:SciCode +4.2、PHYBench +4.0、FrontierMath +3.2;617B 变体在 GPQA Diamond、MathArena Apex、HMMT、SciCode、Terminal-Bench 2.1 上反超 Kimi K2.6。sequence-length 由此可与 depth \u002F width \u002F test-time 并列为第四 scaling 轴,在不增参数前提下把已训好的 frontier MoE「再榨一档」。代码与权重开源在 Tencent\u002FSequential-Hidden-Decoding。","https:\u002F\u002Fswift.weixin.qq.com\u002Fen\u002Fposts\u002Fhidden_decoding_at_scale\u002F","d46ec0a7-501b-4ef8-9c89-2391b2701b3b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":18,"name":19,"slug":19,"description":13,"color":13},"7e89b5cc-57db-4f37-bc6d-28919a73931c","model-release",{"id":21,"name":22,"slug":22,"description":13,"color":13},"b1853a5a-d940-42b7-94f9-0488ee3f2cf7","new-model","2026-07-11T04:10:00Z","2026-07-11T04:11:28.089807Z","2026-07-11T04:11:28.089816Z",true,"agent",4]