[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-llm-linear-superposition-two-thoughts":3,"topics-all":38,"news-related-4c12b881-77bd-48a5-99ca-1e03152ab7ca":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"4c12b881-77bd-48a5-99ca-1e03152ab7ca","LLM叠加态实测:两段文本向量平均后,模型同时接住两条线","把两段无关文本的embedding逐位平均后喂给预训练LLM,输出不是乱码:两段的下一个token都还在分布前排,top-10命中约三到四成。论文称之为线性叠加假说:它随预训练推进而衰减,用不到原数据0.025%的微调即可大幅恢复。","Transformer 里塞满非线性组件——softmax 注意力、非线形激活的 MLP,主流认知因此默认:想同时处理多条独立文本流,要么分开跑多次前向,要么改架构防止互相干扰。一篇 9 月 24 日提交 arXiv 的论文([arXiv:2609.29845](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.29845))给出了相反的实证:把两段文本的 embedding 逐位平均,预训练 LLM 的输出近似两条文本下一个 token 分布的叠加。作者把它命名为**叠加线性假说**(Superposition Linearity Hypothesis)。\n\n## 信号没被淹没,反而站在前排\n\n研究者在 Pythia、Qwen、Llama、OLMo、Gemma 五个家族的预训练模型上测:两段无关文本的 embedding 混合后,单流预测 token 有约 30-40% 仍排进混合分布 top-10,top-50 是 50-60%,top-100 达 60-65%。词表至少 5 万起步,这个恢复率远高于噪声底。\n\n需要警惕的是「高频词侥幸」解释:随机词频基线下跨流 ground-truth token 落进 top-10 只有 2.63%、top-100 只有 10.41%,而叠加前向的 top-10 恢复率比基线高一个数量级——效应来自叠加结构本身,不是 Zipf 分布的假象。\n\n## 训练越久,叠加越弱\n\n最反直觉的发现:叠加性在初始化时最强,随预训练单调衰减。也就是说这不是模型学会的能力,而是架构自带的「出厂设置」,训练目标反而在持续侵蚀它。作者用隐藏态可加性误差逐 checkpoint 追踪,发现误差随训练单调增长;层间线性度呈 U 型——深层(约 2L\u002F3 以下)保持近线性,这给叠加信号能活着传到输出 logits 提供了几何解释。\n\n## 轻量微调能把线性拉回来\n\n用不到原预训练数据规模 0.025% 的数据做轻量微调,能把混合分布与理想叠加目标的散度大幅压低。代价写得坦白:同一目标会损伤单流质量,Pythia-2.8B 在 LAMBADA 上从 0.544 掉到 0.357,Qwen2.5-3B 从 0.602 掉到 0.460。算力上不算白给:Pythia-2.8B 微调 15 万步用了 2×A100 80GB 约 114 小时。\n\n## 从一次前向拆出两条续写\n\n几何平均阻碍是核心难点:叠加 logits 下概率正比于两流概率的几何平均,任何一流的高概率 token 都会被另一流压低。Joint Contrastive 解码用一个小模型做逐流引导,把 Llama-3.2-3B 的混合前向 LAMBADA 均值从 0.182 拉到 0.430(小模型单流基线 0.540),流间 Jaccard 重叠压到 0.067。结论一节称,若拆解彻底,理论上单次前向可产出两条连贯续写、吞吐 2 倍、每活跃流 KV cache 减半。\n\n## 所以呢\n\n这篇论文把「并行推理」的想象空间和它的现实边界同时摆上了桌:正面的部分,叠加是架构内生的、可恢复的、可拆解的;清醒的部分,LAMBADA 0.430 对 0.540 的差距、单流质量的牺牲、L≤512 的短上下文,都说明这还是 proof of concept。对比 DataMUX、Superposed Decoding 等需要外加 mux\u002Fdemux 结构的方案,它的差异点是「叠加不需要被工程强加,它本来就在」。对推理成本敏感的团队,这条线索值得盯:如果拆解损失能进一步收窄,2 倍吞吐不是营销话术,是结构性的。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.29845","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"1f89222f-c1bf-4d01-8c4d-c3349b816255","en","Your Transformer Can Hold Two Thoughts at Once: LLM Superposition","Average two texts' embeddings and LLMs output a superposition of both next-token distributions — pretraining erodes it, tiny fine-tuning restores it.","Transformers are packed with non-linear components — softmax attention, MLP blocks with non-linear activations — so the prevailing assumption is that processing multiple independent text streams requires separate forward passes or architecture changes to avoid destructive interference. A paper submitted to arXiv on Sep 24 ([arXiv:2609.29845](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.29845)) offers counterevidence: when the embeddings of two texts are averaged token-wise, a pretrained LLM's output approximates a superposition of the two individual next-token distributions. The authors name this the **Superposition Linearity Hypothesis**.\n\n## The signal survives, near the top\n\nThe researchers evaluated pretrained models across five families — Pythia, Qwen, Llama, OLMo, Gemma. After mixing embeddings of two unrelated texts, the single-stream predicted token still lands in the mixed distribution's top-10 in roughly 30-40% of cases, top-50 in 50-60%, and top-100 in 60-65%. With vocabularies of at least 50,000 tokens, this recovery rate sits far above the noise floor.\n\nThe \"lucky high-frequency token\" explanation does not hold: under a random frequency baseline, the cross-stream ground-truth token reaches top-10 in only 2.63% and top-100 in 10.41% of cases, while the superposed forward pass exceeds the baseline by an order of magnitude — the effect comes from the superposition structure itself, not a Zipf artifact.\n\n## The longer it trains, the weaker it gets\n\nThe most counterintuitive finding: superposition fidelity is strongest at initialization and decays monotonically as pretraining proceeds. This is not a capability the model learns; it is a factory setting of the architecture that the training objective steadily erodes. The authors track hidden-state additivity error checkpoint by checkpoint and find it grows monotonically. Layer-wise linearity shows a U-shaped profile — deep layers (beyond roughly 2L\u002F3) stay near-linear — offering a geometric explanation for why the superposed signal survives to the output logits.\n\n## Lightweight fine-tuning pulls linearity back\n\nFine-tuning on less than 0.025% of the original pretraining data substantially reduces the divergence between the mixed-input distribution and the ideal superposition target. The cost is stated plainly: the same objective measurably hurts single-stream quality — Pythia-2.8B drops from 0.544 to 0.357 on LAMBADA, Qwen2.5-3B from 0.602 to 0.460. Compute is not free either: the Pythia-2.8B run of 150,000 steps took about 114 hours on 2×A100 80GB GPUs.\n\n## Decoding two continuations from one forward pass\n\nThe geometric-mean obstruction is the core difficulty: under averaged logits, probabilities scale with the geometric mean of the two streams' distributions, so any token highly likely in one stream gets suppressed by the other. Joint Contrastive decoding uses a small auxiliary model for per-stream guidance, lifting Llama-3.2-3B's mixed-pass LAMBADA mean accuracy from 0.182 to 0.430 (small-model single-stream baseline: 0.540) while pressing inter-stream Jaccard overlap down to 0.067. The conclusion states that with full disentanglement, a single forward pass could yield two coherent continuations — theoretically 2x throughput and half the KV-cache footprint per active stream.\n\n## So what\n\nThe paper puts both the promise and the boundaries of parallel inference on the table at once. On the positive side, superposition is intrinsic to the architecture, recoverable, and partially disentangleable. On the sober side, the 0.430-vs-0.540 LAMBADA gap, the single-stream quality sacrifice, and the short L≤512 contexts all mark this as a proof of concept. Compared with DataMUX, Superposed Decoding, and other schemes that bolt on mux\u002Fdemux structure, the difference here is that superposition does not need to be engineered in — it was already there. For teams sensitive to inference cost, this is a thread worth tracking: if the disentanglement loss can be narrowed further, 2x throughput is not marketing talk but structural.","llm-linear-superposition-two-thoughts","2026-09-25T21:09:18Z","2026-09-25T21:09:21.897012Z","2026-09-25T21:09:21.897020Z",true,"agent",1015,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"3d922c00-afcb-4f1c-a6d5-8f9d6c10c642","从 Kimi Linear 到 Kimi K3:MoE 推理效率战里被忽略的架构升级","kimi-k3-latentmoe-kda-attnres-nope","2026-07-30T00:30:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"71d39257-c99a-4970-876a-571a8789bf42","ICLR 2026 杰出论文出炉：两项研究揭示 Transformer 与多轮对话的深层局限","iclr-2026-outstanding-papers-transformer-multiturn","2026-05-03T07:08:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"e40de2f9-9ee4-47f3-b1f1-7bf93a4870a9","一个动词翻转工具调用决策,LLM 内部向量现形","llm-tool-call-decision-vector","2026-10-09T13:10:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"fc533af5-9e8a-43c2-8408-2f931a6aa398","EngramEdit:把事实塞进 LLM 的记忆抽屉","engramedit-decoupled-knowledge-llm","2026-10-09T04:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"282f3cb4-429c-4432-b632-5e7288192eb9","MassAlloc注意力:按质量分配算力,反传快3倍","massalloc-attention-mala","2026-09-29T21:08:59+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"0bdbdcd9-fbb5-4123-9854-57b8f28b385b","把注意力退回查字典:LEMA让KV缓存离开显存","lema-exact-match-attention","2026-09-25T15:15:00+00:00"]