[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-latent-lattice-language-models-horizontal-latent-perplexity":3,"topics-all":38,"news-related-1dec93aa-e2a2-4b65-941c-6fa04d452175":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"1dec93aa-e2a2-4b65-941c-6fa04d452175","Lattice横向潜变量:困惑度15.41降到14.40","MIT CSAIL 与 Adobe 提出 Lattice of Latents:给每层加一条跨 token 存活的横向潜变量流。20B token 预训练下,406M 模型困惑度从 15.41 降到 14.40,追平参数多 1.5 倍的 40 层 Transformer,叠加超连接后到 13.61。","给大模型加算力，主流办法大致三条：加深、循环，或者把残差流本身加宽。MIT CSAIL 与 Adobe 的一篇论文（arXiv:2611.21608，2026 年 11 月 16 日提交，第一作者 Shivam Duggal）问了另一个问题——不是「每个 token 做多少计算」，而是「模型同时维持多少个活跃的计算状态」。\n\n论文的答案叫 Lattice of Latents：在纵向那条贯穿网络深度的残差流之外，给每一层挂一条沿时间横向复现的潜变量流。这条流用仿射递推更新，跨 token 存活，并且直接去调制纵向流的注意力、KV 写入、MLP 计算与解码。\n\n这是它跟混合 Transformer-SSM 架构的关键分野。在那些架构里，循环状态主要承担序列混合的内部记忆；而 Lattice 的横向潜变量不去替代注意力，它是陪伴残差流的「同伴状态」。论文特别指出，它不会增加每 token 的 KV cache 条目——这与加层、加循环步数的思路有本质区别。\n\n## 数据:15.41 到 14.40 这笔账\n\n在 20B FineWeb-Edu token 上预训练，论文报告的 FineWeb-Edu 验证困惑度是这么排的（按论文自己的实测口径）：\n\n- 参数匹配的基线 Transformer Base-24L（406.7M）：15.410\n- LLLM (945, 256)（406.5M）：14.545\n- LLLM (908, 256) 加输入注入与深度读出流（405.6M）：14.400\n- 同一模型改成顺序推理、不做 Jacobi 扫描：14.733\n\n也就是说，406M 级别的横向潜变量模型把困惑度从 15.41 压到 14.40，而这个数字追平了参数量多 1.5 倍的 40 层 Transformer（Base-40L，609.1M，同样是 14.400）。每 token 计算量几乎没变：基线 0.71 GFLOP\u002Ftoken，Lattice 是 0.72。\n\n提升更大的是带非线性自顶向下反馈的变体。论文用三次 Jacobi 扫描近似这条反馈路径，训练时 FLOPs 约为 2.6 倍，困惑度降到 14.026；推理时改回顺序展开，计算量回到 0.72，困惑度则是 14.733。这一档的取舍很直白：算力换精度，或者反过来。\n\n## 叠加:13.61 与 13.76 口径不一致\n\nLattice 的卖点之一是它跟既有的残差流增强正交，可以叠加。论文在同规模（400M 参数、20B token）下测了一圈：单独用 Lattice 把 15.41 降到 14.47，优于超连接的 15.28、流形约束超连接的 14.93 和 value residual 的 14.69，但略逊于输入残差的 14.34。\n\n真正有意思的是叠加：加上 Lattice 后，超连接从 15.28 降到 14.21（降幅 7.0%），超连接加输入残差从 14.22 降到 13.61（降幅 4.3%）。论文把这一组排在它测过的全部配置的第一位，称它比原始 Transformer 低 11.7%，比纯 Transformer 组合里输入残差加 value residual 那一档（14.16）还低 3.9%。\n\n这里有个细节值得记一笔：论文摘要和 4.2 节正文写的都是 13.61，但图 3 的标题写的是「Lattice + HC + x0 reaches 13.76, the best overall」。附录另有一处写「learned value residuals 对 Lattice 几乎没有作用（13.61 → 13.62）」。三个数说的是同一个组合，对不上。引用时以正文为准更稳妥，但这个不一致本身说明论文的校对流程还没走完。\n\n## 数学推理:同一个困惑度，差 10.1 分\n\n困惑度之外的账在 GSM8K 上。论文在 GSM8K-Aug（385,620 道题）上做思维链微调，400M 规模下，基线 Transformer 的五项平均准确率是 36.02，带 Jacobi 的 LLLM 是 48.10，提升 12.08 分；600M 规模上从 40.24 提到 47.74，提升 7.50 分。论文的说法是，两个规模上每一个 Lattice 变体都比每一个 Transformer 基线高出至少 10.1 个 GSM8K 分。\n\n更能说明问题的一组对照是：Base-40L 与 LLLM (908, 256) 的预训练困惑度完全相同，都是 14.40，零样本平均也几乎一样（47.36 对 47.38），但 GSM8K 上差了 10.1 分（33.5 对 43.6）。同一个起点，微调之后拉开十个百分点——说明预训练困惑度并不决定微调后的推理能力，中间还夹着一层这个指标捕捉不到的东西。\n\n## 循环骨干:4 倍状态差的反超\n\n论文还把 Lattice 装到了 Gated DeltaNet、Mamba-2、Triadic GDN 和 GDN-Mamba-2 混合模型上（10B token，约 405M 参数），横向潜变量分别调制这些骨干的记忆读取、写入与更新。结果是在每一个测试过的原生状态尺寸上都降了困惑度；其中 Lattice-GDN 与 Lattice-Triadic-GDN 超过了持有 4 倍原生状态条目的纯基线。\n\n下游九任务平均，30 组配对里 Lattice 领先 29 组；只有一次回退，是混合模型在 128K 状态、五样本设置下的 45.02 掉到 44.80。\n\n更硬的一层证据来自对 GDN 循环记忆本身的探测：在匹配状态尺寸下，Lattice 模型的记忆更集中（rent 降低 9.6% 到 18.3%），查询与主导记忆方向的对齐差距从 16K 时的 +1.89 拉宽到 256K 时的 +5.36，对齐倍数从纯模型的 8.0 倍提到 13.3 倍。\n\n## 该泼的冷水\n\n三件事必须一起看。\n\n第一，规模。这批实验全部在 400M 到 600M 参数、10B 到 30B token 上完成，2048 上下文，LLaMA 式解码器配 GPT-NeoX 分词器。论文自己把结论限定在「测试过的配置里」，没有往更大规模外推。\n\n第二，复现性。GSM8K 微调每个模型只跑了一个随机种子（seed 0），论文明确说不解读 Lattice 各变体之间的细小差异，只能解读与 Transformer 基线之间更大且一致的那道差距。循环骨干那部分的耦合方式还是事后挑的，论文自己提示增益应当读作「最佳耦合方式」的增益，而不是某个预先选定方式的增益。\n\n第三，代码。项目页面上 Paper 与 Code 两个链接此刻都写着 coming soon，想上手的人现在还拿不到实现（项目页：https:\u002F\u002Fshivamduggal4.github.io\u002Flatent-lattice-language-models\u002F）。\n\n## 所以呢\n\n这篇论文值得带走的不是那 14.40，而是它把一个大家默认的前提摆到了台面上：扩展算力不只有「让每个 token 干更多活」这一条路，也可以走「让模型同时手握更多条活」。横向潜变量是一条独立于深度的算力轴，而且能和既有的残差流工程正交叠加。\n\n对还在堆深度、加 KV 条目的团队来说，这提示了一笔此前算得少见的账：在不增加每 token KV cache 条目的前提下，提高每层维持的活跃状态数。至于这些数字能不能撑到真实的千亿参数规模，得等代码放出来，也得等别人在更大规模上重跑一遍。论文本身：https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.21608\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.21608","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"658b83fa-a9e6-401c-a503-bb15c5fbf9b4","en","Horizontal Latents Cut Perplexity From 15.41 To 14.40","MIT and Adobe add a per-layer latent stream. At 406M parameters and 20B tokens, perplexity falls from 15.41 to 14.40, matching a 1.5x larger model.","The standard ways to add computation to a language model are roughly three: go deeper, loop, or widen the residual stream itself. A paper from MIT CSAIL and Adobe (arXiv:2611.21608, submitted 16 November 2026, first author Shivam Duggal) asks a different question. Not how much computation each token gets, but how many active computational states the model maintains at the same time.\n\nThe answer is called the Lattice of Latents. Alongside the vertical residual stream that runs through network depth, the method attaches a horizontal latent stream to every layer, running recurrently along time. That stream updates through an affine recurrence, persists across tokens, and directly conditions the vertical stream's attention, KV-cache writes, MLP computation, and decoding.\n\nThis is where it separates from hybrid Transformer-SSM architectures. In those, the recurrent state mainly serves as internal memory for sequence mixing. The Lattice's horizontal latent does not replace attention; it is a companion state that sits beside the residual stream. The paper stresses that it adds no per-token KV-cache entries, which is a categorically different bet from adding layers or looped reasoning steps.\n\n## The numbers: from 15.41 to 14.40\n\nAfter pretraining on 20B FineWeb-Edu tokens, the paper reports FineWeb-Edu validation perplexity as follows, in its own measured terms:\n\n- Parameter-matched baseline Transformer Base-24L (406.7M): 15.410\n- LLLM (945, 256) (406.5M): 14.545\n- LLLM (908, 256) with input injection and depth-wise readout stream (405.6M): 14.400\n- The same model rolled out sequentially, without Jacobi sweeps: 14.733\n\nSo a 406M-scale horizontal-latent model pushes perplexity from 15.41 down to 14.40, and that number matches a 40-layer Transformer with 1.5 times more parameters (Base-40L, 609.1M, also 14.400). Per-token compute barely moves: the baseline is 0.71 GFLOP\u002Ftoken, the Lattice 0.72.\n\nThe variant with nonlinear top-to-bottom feedback goes further. The paper approximates that feedback path with three Jacobi sweeps, costing about 2.6 times the FLOPs during training, and reaches perplexity 14.026. At inference the same model can instead be rolled out sequentially at the base FLOPs, and the perplexity becomes 14.733. The trade is plain: spend compute for accuracy, or the other way around.\n\n## Composing: 13.61 and 13.76 do not agree\n\nOne selling point is that the Lattice is orthogonal to existing residual-stream enhancements and can be stacked with them. The paper runs a comparison at matched scale (400M parameters, 20B tokens). On its own the Lattice takes 15.41 down to 14.47, ahead of hyper-connections at 15.28, manifold-constrained hyper-connections at 14.93, and value residuals at 14.69, though behind input residuals at 14.34.\n\nThe interesting part is the stacking. With the Lattice added, hyper-connections fall from 15.28 to 14.21 (a 7.0% drop), and hyper-connections plus input residuals fall from 14.22 to 13.61 (a 4.3% drop). The paper puts that combination at the top of the configurations it tested, calling it 11.7% below the plain Transformer and 3.9% below the pure Transformer composition of input residuals plus value residuals (at 14.16).\n\nOne detail is worth recording. The abstract and Section 4.2 both say 13.61, but the Figure 3 caption says 'Lattice + HC + x0 reaches 13.76, the best overall'. A passage in the appendix separately states that learned value residuals do little for the Lattice (13.61 to 13.62). Three numbers for the same combination, and they do not line up. Quoting 13.61 from the body text is the safer bet, but the mismatch itself suggests the paper has not been through a full proofread.\n\n## Math reasoning: same perplexity, 10.1 points apart\n\nThe bill beyond perplexity comes due on GSM8K. The paper finetunes each 20B-token checkpoint on GSM8K-Aug (385,620 problems) with chain-of-thought. At 400M scale, the baseline Transformer's five-benchmark mean accuracy is 36.02, while the Jacobi-refined LLLM reaches 48.10, a gain of 12.08 points. At 600M scale, the mean rises from 40.24 to 47.74, a gain of 7.50 points. The paper's claim is that at both scales every Lattice variant beats every Transformer baseline by at least 10.1 GSM8K points.\n\nThe more telling pair of numbers: Base-40L and LLLM (908, 256) have exactly the same pretraining perplexity, 14.40, and nearly the same zero-shot average (47.36 against 47.38), yet they differ by 10.1 points on GSM8K (33.5 against 43.6). Same starting point, a ten-point gap after finetuning. Pretraining perplexity therefore does not determine finetuned reasoning ability; something between the two is not captured by that metric.\n\n## Recurrent backbones: beating baselines with 4x the state\n\nThe paper also mounts the Lattice on Gated DeltaNet, Mamba-2, Triadic GDN, and GDN-Mamba-2 hybrids (10B tokens, roughly 405M parameters), with the horizontal latents modulating each backbone's memory read, write, and update. Perplexity improves at every native state size tested, and both Lattice-GDN and Lattice-Triadic-GDN outperform pure baselines that hold 4 times as many native-state entries.\n\nOn the nine-task downstream average, the Lattice leads in 29 of 30 pairs. The only regression is the hybrid at 128K state in the five-shot setting, dropping from 45.02 to 44.80.\n\nA harder layer of evidence comes from probing GDN's recurrent memory directly. At matched state sizes, Lattice models hold more concentrated memory (rent falls by 9.6% to 18.3%), and the alignment gap between queries and the leading memory direction widens from +1.89 at 16K to +5.36 at 256K, with alignment reaching 13.3 times chance against 8.0 times for the pure model.\n\n## Three caveats\n\nThree things have to be read alongside the results.\n\nFirst, scale. Every experiment here runs at 400M to 600M parameters and 10B to 30B tokens, with a 2048-token context, a LLaMA-style decoder, and the GPT-NeoX tokenizer. The paper limits its own conclusion to the configurations tested, and does not extrapolate to larger scale.\n\nSecond, reproducibility. Each GSM8K finetune runs a single seed (seed 0) per model. The paper explicitly declines to interpret the small differences among Lattice variants, offering only the larger and consistent gap to the Transformer baselines. The coupling used for the recurrent backbones was also selected post hoc, and the paper notes the gains should be read as those of the best coupling rather than of any single preselected one.\n\nThird, code. On the project page, both the Paper and Code links currently read 'coming soon', so there is no implementation to work with yet (project page: https:\u002F\u002Fshivamduggal4.github.io\u002Flatent-lattice-language-models\u002F).\n\n## So what\n\nWhat is worth carrying away from this paper is not the 14.40. It is that the paper puts an assumed premise on the table: scaling computation does not have to mean making each token do more work. It can also mean letting the model hold more things at once. A horizontal latent is an axis of computation independent of depth, and it compounds with the residual-stream engineering that already exists.\n\nFor teams still stacking depth or adding KV-cache entries, this points at a bill that had rarely been calculated carefully: raising the number of active states each layer maintains, without adding per-token KV-cache entries. Whether these numbers survive at real hundred-billion-parameter scale depends on the code landing, and on other people rerunning the experiment at larger sizes. The paper itself: https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.21608\n","latent-lattice-language-models-horizontal-latent-perplexity","2026-11-17T09:20:00Z","2026-11-17T09:09:31.036508Z","2026-11-17T09:09:31.036522Z",true,"agent",160,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"384a7e0d-7cca-4051-be56-51c042c8f7b8","参数向量也能king-man+woman:快速学习的几何答案","parameter-space-analogies-rapid-learning","2026-11-15T17:10:44+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"f6d63800-dab5-4965-9d33-402a69d2c121","先动手还是先打主意:LLM的目标藏在注意力头里","llm-goal-encoding-attention-heads","2026-11-14T01:05:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"27b8351d-d0f7-4e89-b5bc-ad3a3fce9a50","不跑模型不喂数据:一个标量预测归纳头","rho-invariant-predicts-induction-heads","2026-11-10T21:20:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"fc533af5-9e8a-43c2-8408-2f931a6aa398","EngramEdit:把事实塞进 LLM 的记忆抽屉","engramedit-decoupled-knowledge-llm","2026-10-09T04:00:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"cb7fb8b3-5862-4cba-adab-c4794e989966","图灵奖得主 Pearl 长访谈：LLM 能讲因果只是因为人类替它爬过了因果阶梯","judah-pearl-llm-causal-ladder-agi","2026-07-31T07:00:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"f8436dd3-d6fc-4ea7-9f2e-1086026c11d0","Transformer 的几何之眼：arXiv 2607.17146 把注意力炼成薛定谔桥，把 SGD 写成伊藤扩散","transformer-geometry-schrodinger-bridge","2026-07-23T12:10:00+00:00"]