Tencent: sequence length as LLM's 4th axis, WeLM-HD4 beats K2.6

Tencent's WeChat AI / WeLM team on July 9 released "Hidden Decoding at Scale", making sequence length a new fixed-backbone extension axis. Through multi-stream embedding expansion (n=4) + Stream-Factorized Attention, continued pretraining is done on WeLM-80B and WeLM-617B MoE, with 9 benchmarks all beating their own AR baselines; the 617B variant surpasses Kimi K2.6 on GPQA Diamond, PHYBench, MathArena Apex, HMMT, and SciCode, while training cost grows only near-linearly (80B 5.1× / 617B 4.4×). The Tencent WeChat AI / WeLM team on July 9 released "Hidden Decoding at Scale" on arXiv, pushing the "latent computation scaling" idea to the 100B+ MoE scale. The method doesn't do depth/width expansion, but instead copies each token's embedding n times and expands it along the sequence direction, doing a single Transformer backbone forward at once, with only the final stream connected to the LM head loss, the front n-1 streams acting as latent scratchpad without supervision — backbone parameters unchanged, each token gaining n times effective compute in a single forward. The key to making the method run on 100B+ is Stream-Factorized Attention: most layers only do within-stream causal attention, with a few layers doing cross-stream mixing, compressing the n² growth to near-linear. The 80B and 617B training single-step times are 5.1× and 4.4× of the un-extended baseline, falling between "ideal 4×" and "fully connected 16×". Stacked with WeLM backbone's own KV-mirror design (back-half layer KV only depends on front-half layer hidden state), mirror layers only run the final stream, with 80B 32k single batch saving another 20%. In effect, HD4-80B and HD4-617B are uniformly positive over their own AR baseline on 9 benchmarks under the same CPT + early SFT-only protocol: SciCode +4.2, PHYBench +4.0, FrontierMath +3.2; the 617B variant surpasses Kimi K2.6 on GPQA Diamond, MathArena Apex, HMMT, SciCode, and Terminal-Bench 2.1. Sequence-length can thus be listed as the fourth scaling axis alongside depth / width / test-time, "squeezing one more tier" from a trained frontier MoE without adding parameters. Code and weights are open-sourced at Tencent/Sequential-Hidden-Decoding.