[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-tencent-welm-hd4-617b-sequence-scaling":3,"topics-all":36,"news-related-224fc205-6b1a-48d3-8562-df66921d017e":55},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"224fc205-6b1a-48d3-8562-df66921d017e","腾讯把序列长度做成 LLM 第四 scaling 轴：WeLM-HD4-617B 在不增参数前提下反超 Kimi K2.6","腾讯 WeChat AI \u002F WeLM 团队 7 月 9 日发布《Hidden Decoding at Scale》,把序列长度作为新的固定 backbone 扩展轴,通过多流嵌入展开(n=4)+ Stream-Factorized Attention,在 WeLM-80B 和 WeLM-617B 两个 MoE 上做 continued pretraining,9 项基准全胜自身 AR 基线,617B 变体在 GPQA Diamond、PHYBench、MathArena Apex、HMMT、SciCode 上反超 Kimi K2.6,训练成本仅近线性增长(80B 5.1× \u002F 617B 4.4×)。\n\n腾讯 WeChat AI \u002F WeLM 团队 7 月 9 日在 arXiv 放出《Hidden Decoding at Scale》,把「latent computation scaling」思路推到 100B+ MoE 体量。方法不做深度\u002F宽度扩展,而是把每个 token 的 embedding 复制 n 份沿序列方向展开,在同一 Transformer backbone 一次性前向,只有最终流(stream)接 LM head 损失,前面 n-1 流当 latent scratchpad 不受监督——backbone 参数不变,每个 token 在一次前向里获得 n 倍有效计算。\n\n让方法能跑上 100B+ 的关键是 Stream-Factorized Attention:大多数层只做流内因果注意力,少数层做跨流混合,把 n² 增长压到近线性。80B 和 617B 训练单步时间为未扩展基线的 5.1× 和 4.4×,落在「理想 4×」与「全连接 16×」之间。再叠 WeLM backbone 自带的 KV-mirror 设计(后段层 KV 只依赖前段层 hidden state),镜像层只跑最终流,80B 32k 单 batch 再省 20%。\n\n效果上 HD4-80B 与 HD4-617B 在相同 CPT + 早期 SFT-only 协议下 9 项基准相对自身 AR 基线全部正向:SciCode +4.2、PHYBench +4.0、FrontierMath +3.2;617B 变体在 GPQA Diamond、MathArena Apex、HMMT、SciCode、Terminal-Bench 2.1 上反超 Kimi K2.6。sequence-length 由此可与 depth \u002F width \u002F test-time 并列为第四 scaling 轴,在不增参数前提下把已训好的 frontier MoE「再榨一档」。代码与权重开源在 Tencent\u002FSequential-Hidden-Decoding。","https:\u002F\u002Fswift.weixin.qq.com\u002Fen\u002Fposts\u002Fhidden_decoding_at_scale\u002F","d46ec0a7-501b-4ef8-9c89-2391b2701b3b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":18,"name":19,"slug":19,"description":13,"color":13},"7e89b5cc-57db-4f37-bc6d-28919a73931c","model-release",{"id":21,"name":22,"slug":22,"description":13,"color":13},"b1853a5a-d940-42b7-94f9-0488ee3f2cf7","new-model",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"83450268-d71d-4f10-a326-40e454f37e2b","en","Tencent: sequence length as LLM's 4th axis, WeLM-HD4 beats K2.6","Tencent's WeChat AI \u002F WeLM team on July 9 released \"Hidden Decoding at Scale\", making sequence length a new fixed-backbone extension axis. Through multi-stream embedding expansion (n=4) + Stream-Factorized Attention, continued pretraining is done on WeLM-80B and WeLM-617B MoE, with 9 benchmarks all beating their own AR baselines; the 617B variant surpasses Kimi K2.6 on GPQA Diamond, PHYBench, MathArena Apex, HMMT, and SciCode, while training cost grows only near-linearly (80B 5.1× \u002F 617B 4.4×). The Tencent WeChat AI \u002F WeLM team on July 9 released \"Hidden Decoding at Scale\" on arXiv, pushing the \"latent computation scaling\" idea to the 100B+ MoE scale. The method doesn't do depth\u002Fwidth expansion, but instead copies each token's embedding n times and expands it along the sequence direction, doing a single Transformer backbone forward at once, with only the final stream connected to the LM head loss, the front n-1 streams acting as latent scratchpad without supervision — backbone parameters unchanged, each token gaining n times effective compute in a single forward. The key to making the method run on 100B+ is Stream-Factorized Attention: most layers only do within-stream causal attention, with a few layers doing cross-stream mixing, compressing the n² growth to near-linear. The 80B and 617B training single-step times are 5.1× and 4.4× of the un-extended baseline, falling between \"ideal 4×\" and \"fully connected 16×\". Stacked with WeLM backbone's own KV-mirror design (back-half layer KV only depends on front-half layer hidden state), mirror layers only run the final stream, with 80B 32k single batch saving another 20%. In effect, HD4-80B and HD4-617B are uniformly positive over their own AR baseline on 9 benchmarks under the same CPT + early SFT-only protocol: SciCode +4.2, PHYBench +4.0, FrontierMath +3.2; the 617B variant surpasses Kimi K2.6 on GPQA Diamond, MathArena Apex, HMMT, SciCode, and Terminal-Bench 2.1. Sequence-length can thus be listed as the fourth scaling axis alongside depth \u002F width \u002F test-time, \"squeezing one more tier\" from a trained frontier MoE without adding parameters. Code and weights are open-sourced at Tencent\u002FSequential-Hidden-Decoding.","tencent-welm-hd4-617b-sequence-scaling","2026-07-11T04:10:00Z","2026-07-11T04:11:28.089807Z","2026-08-19T02:08:40.142862Z",true,"agent",405,[37,46],{"slug":38,"tag_slug":38,"title_zh":39,"title_en":40,"intro_zh":41,"intro_en":42,"id":43,"is_active":33,"created_at":44,"modified_at":45},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":47,"tag_slug":47,"title_zh":48,"title_en":49,"intro_zh":50,"intro_en":51,"id":52,"is_active":33,"created_at":53,"modified_at":54},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":56},[57,62,67,72,77,82],{"id":58,"title":59,"news_slug":60,"published_at":61},"b667e52f-ec7d-4ca4-8d9e-1db81e1a5616","DeepSeek论文:890字节KV缓存的三层架构账","deepseek-v41-flash-kv-cache-paper","2026-09-18T15:10:00+00:00",{"id":63,"title":64,"news_slug":65,"published_at":66},"d056f67b-7e0d-4e44-8d39-e31ea50deeae","Bonsai 2 27B 三元压缩:Qwen3.8 压到 5.9 GB,benchmark 留存 98.2%","bonsai-2-27b-ternary-qwen3-8-compression","2026-09-17T15:47:00+00:00",{"id":68,"title":69,"news_slug":70,"published_at":71},"4c4a2a9e-f69b-4985-bd42-97ab2ef4e2ac","Spark-X2.5-4B 开源:4B 跑 1M 上下文,22 项基准打 9B 级 Qwen3.5","spark-x2-5-4b-apache-open-source","2026-09-16T01:30:00+00:00",{"id":73,"title":74,"news_slug":75,"published_at":76},"ee62535f-b897-437b-8674-02801632dadb","DeepSeek V4.1 非对称架构首发:读题 8B 答题 16B,KV 缓存砍到初代的 1\u002F437","deepseek-v4-1-flash-ced-kv-cache","2026-09-10T15:10:00+00:00",{"id":78,"title":79,"news_slug":80,"published_at":81},"ee700812-e4e5-4fa5-8687-6a0d7e5c7f78","Mercury 2.5 发布：扩散 LLM 跑出 1107 tokens\u002F秒，智能较上代提升 40%","mercury-2-5-diffusion-llm","2026-09-09T17:05:00+00:00",{"id":83,"title":84,"news_slug":85,"published_at":86},"491f4904-c854-4925-b3e3-e34b8afd5e50","KDA+MLA 混合栈下沉到 1.3B 激活:Ling-3.0-tiny 把 MoE 端侧化,INT4 跑出 115 tok\u002Fs","ling-3-tiny-kda-mla-edge-deployment","2026-08-18T00:00:00+00:00"]