[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-context-rot-attention-competition-not-length":3,"topics-all":38,"news-related-a43599f1-d635-4174-9a25-6e1031c497b1":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"a43599f1-d635-4174-9a25-6e1031c497b1","长上下文崩掉的真凶不是长度，是内容竞争","长上下文真的是越长越好吗？论文用五个开源模型做检索试验给出否定答案：中性填充物塞进上下文几乎不掉分，换成同格式的矛盾陈述会让准确率从 0.98 掉到 0.11；残留在证据上的注意力质量能解释 35% 到 58% 的损失方差，而长度本身最多解释 2% 到 8%。","大模型圈最顺手的解释是：上下文越长，模型越笨，这看起来已经是常识。2025 年 Chroma 的技术报告在 18 个前沿模型上把这种退化命名为 \"context rot\"，而 Du 等人那篇《Context Length Alone Hurts LLM Performance Despite Perfect Retrieval》给出了更干净的证据：把无关 token 从注意力里屏蔽掉、只保留位置索引，模型在数学、问答、代码任务上照样掉 13.9% 到 85%。既然把干扰全藏起来还会掉，那长度本身就是元凶。\n\n2026 年 11 月 3 日，UC San Diego 的 Yinuo Liu 在 arXiv 上放出正好冲着这个结论来的一篇论文（[arXiv:2611.04169](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.04169)）。设计只有一条原则：所有对比都锁死长度和证据位置，只换证据周围填的东西。五个开源模型、0.5B 到 3B、四个模型家族，任务是键值检索——上下文里藏一句「The secret code of the river is 4821.」，结尾问河的密码是多少，算正确值在 teacher forcing 下的对数概率。四种条件：什么都不填、填中性内容（随机数字、随机词、一句重复话）、填同格式的竞争性陈述（键从 51 个普通名词里抽）、以及把填充物从注意力里屏蔽掉但保留位置。\n\n## 长度几乎不要钱，竞争才要命\n\n在 Qwen2.5-1.5B 上，把中性填充物留在上下文里，8K 到 32K 每个长度档的中位损失都压在 0.006 nats 以内，准确率 0.96 到 1.00；同样长度换成竞争性陈述，中位损失直接跳到 2.82、3.94、4.03、4.76 nats，准确率掉进 0.11 到 0.21。让检索崩掉的从来不是那几万 token 本身，而是里面有没有跟证据抢注意力的话。\n\n更狠的对照是把冲突和格式拆开。论文把「共享查询键、但给出不同值」的陈述单独拎出来：五个模型上，不含任何同键冲突的上下文平均损失 0.02 到 0.34 nats，至少含一条的则是 1.38 到 3.24 nats；准确率一边 0.97 到 1.00，一边 0.25 到 0.52。在 2,048 token 固定长度上，同键冲突从 0 条加到 6 条以上，Qwen2.5-1.5B 的损失沿着 0.03、1.37、2.24、2.48、2.71、3.26、3.45 nats 一路爬。反过来，只共享格式、不共享键的干扰项，能解释的损失方差不超过 0.3%。\n\n作者自己在仓库里交代了一个意外：生成竞争填充物时，键是从 51 个词里抽的，没排除被查询的那个键，于是大约每 51 句填充里就有一句会用另一个值重述被问的键——上下文自己跟自己打架。作者写得很直白，这是 accident。仓库里有修掉这问题的 measure_v4.py，加了受控的单冲突实验臂，但它没被跑过，仓库里也没有任何 v4 结果。\n\n## 抢走多少注意力，决定了掉多少分\n\n论文把「注意力还剩多少在证据上」这个量接进回归：在短上下文里挑出对证据注意力最大的八个层头对，读出最后一个 query token 打在证据上的注意力质量 a*。126,473 次试验跑下来，log(1\u002Fa*) 能解释损失方差的 35% 到 58%，而 log 长度最多解释 2% 到 8%。逐模型看，Qwen2.5-1.5B 是 0.58 对 0.05，OLMo-2-1B 是 0.35 对 0.02，Llama-3.2-3B 是 0.45 对 0.08。\n\n还有两个细节值得记。证据上的注意力从短上下文的 0.51 到 0.77，掉到竞争填充下的 0.17 到 0.31；而且这条关系有拐点——在 Qwen2.5-1.5B 上，损失在 a* 掉到大约 0.35 之前几乎不涨，越过这个点之后陡然上去。模型能扛住丢掉一半注意力，但过了线就断崖式崩掉。\n\n## 掩码不是长度的干净对照\n\n这套设计最狠的一刀，是回头去审 Du 等人那条「屏蔽掉干扰也照样掉」的证据。论文把那个掩码协议原样复现到 MMLU：1,000 道题，插入 3,750 到 30,000 个被屏蔽的 token，Llama-3.2-3B 从 51.7% 掉到 40.5%，Qwen2.5-7B 从 65.8% 掉到 48.1%。复现成立，说明这不是某一个模型家族的事。\n\n但同样的掩码丢进检索任务，结论就反了。32K token 处，被屏蔽的中性填充让准确率掉 46 分（0.98 掉到 0.52），而一模一样的填充物留在可见状态，一分不掉；三种中性填充平均下来，可见时 0.96 到 0.99，掩码后 0.48 到 0.54。论文的解释是，掩码留下的是训练里从不出现的长位置空洞，不是「无关 token」——它量出来的代价不是长度的代价。\n\nMMLU 上也有反例：Qwen2.5-1.5B 在 4,096 token 处，不填 59.1%，填可见中性内容 57.5%，填被掩码的 53.9%；但填空格的话，直接掉到 36.0%。所以长度本身也不是完全无辜，只是它在推理任务上留下的伤，和它在检索任务上留下的伤，不是同一道。\n\n## 所以呢\n\n如果这份结论站得住，工程优先级就得反过来：上下文真正的预算不是 token 数，而是「有多少条在跟证据抢话」的陈述数。对检索增强、Agent 记忆、长文档流水线来说，入库前的去重和冲突消解，比把上下文截短更值得做。而且 a* 只需要一次前向传播就能算出来，可以当成上线前的体检指标——抢注意力抢过某个阈值，就该先去处理冲突，而不是扩容窗口。\n\n边界论文自己也划清楚了：所有结论来自单一合成检索任务，模型最大只到 3B；a* 和损失是在同一次前向里量出来的，没有任何因果干预，所以它是预测关系而不是因果结论；冲突分析没有预注册；4,096 token 以上的上下文必然含至少一条冲突，所以 2K 以上的「纯格式竞争者」这条分支根本没被测到。上面那个每 51 句就撞一次键的意外，作者也承认是事故，真正的对照实验还躺在仓库里没跑。\n\n所以更准确的说法不是「长度无害」，而是：在检索里，长度的账要重算，竞争才是那笔要付的钱。谁能把这笔账算清楚，谁就能解释为什么同样标称的上下文窗口，有的模型能用，有的模型一上手就废。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.04169","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"7b62dc40-e003-44db-bbef-73c22be30e3f","en","Attention predicts context rot: competition, not length","rival statements drop accuracy from 0.98 to 0.11, attention left on the evidence explains 35-58% of the loss variance, length 2-8%.","The convenient explanation that circulated through the model community for two years went like this: longer context, dumber model. It had hardened into a consensus. A 2025 Chroma technical report named the degradation \"context rot\" across 18 frontier models, and an earlier paper, \"Context Length Alone Hurts LLM Performance Despite Perfect Retrieval\" by Du et al., offered a clean and convincing proof: mask every irrelevant token out of attention while keeping position indices, and models still degrade by 13.9% to 85% on math, question answering, and coding. If hiding the distraction entirely still costs you, length itself must be the culprit.\n\nOn 3 November 2026, Yinuo Liu of UC San Diego posted a paper aimed squarely at that conclusion ([arXiv:2611.04169](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.04169)). The design rests on one principle: every comparison holds length and evidence position fixed and changes only what surrounds the evidence. Five open models, 0.5B to 3B parameters, four model families. The task is key-value retrieval: the context hides one statement, \"The secret code of the river is 4821.\", and the closing query asks for the river's secret code, scored as the log-probability of the correct value under teacher forcing. Four conditions: nothing filled in, neutral filler (random numbers, random words, one repeated sentence), competing statements in the same template with keys drawn from 51 common nouns, and filler masked out of attention with positions preserved.\n\n## Length is nearly free, competition is not\n\nOn Qwen2.5-1.5B, leaving neutral filler visible keeps the median loss at or below 0.006 nats at every length from 8K to 32K, with accuracy between 0.96 and 1.00. Swapping in competing statements at the same lengths pushes the median loss to 2.82, 3.94, 4.03, and 4.76 nats, and accuracy down to 0.11 to 0.21. What breaks retrieval is never the tens of thousands of tokens themselves. It is whether anything inside them competes with the evidence for attention.\n\nThe sharper control separates conflict from format. The paper isolates statements that share the queried key but assign it a different value: across five models, contexts with no same-key conflict lose 0.02 to 0.34 nats on average, while contexts with at least one lose 1.38 to 3.24 nats. Accuracy is 0.97 to 1.00 on one side and 0.25 to 0.52 on the other. At a fixed length of 2048 tokens, raising same-key conflicts from 0 to 6 or more sends Qwen2.5-1.5B along a climb of 0.03, 1.37, 2.24, 2.48, 2.71, 3.26, and 3.45 nats. In the other direction, distractors that share only the format but not the key explain at most 0.3% of the variance in loss.\n\nThere is one accident the author flags in the repository himself. When generating competing filler, keys were drawn from 51 words without excluding the queried one, so roughly one filler sentence in 51 restates the queried key with a different value, and the context contradicts itself. He puts it plainly: this was an accident. The repository contains a measure_v4.py that fixes the problem and adds a controlled single-rival arm, but it has not been run, and no v4 result appears anywhere in the repository.\n\n## How much attention is stolen decides how much accuracy is lost\n\nThe paper then feeds \"how much attention is left on the evidence\" into a regression. It selects eight layer-head pairs by their attention mass on the evidence within the short context, and reads the attention mass a* that the final query token sends to the evidence tokens. Across 126,473 trials, log(1\u002Fa*) explains 35% to 58% of the variance in loss, while log length explains at most 2% to 8%. Per model, Qwen2.5-1.5B gives 0.58 against 0.05, OLMo-2-1B gives 0.35 against 0.02, and Llama-3.2-3B gives 0.45 against 0.08.\n\nTwo more details are worth keeping. Attention on the evidence falls from 0.51 to 0.77 in the short context to 0.17 to 0.31 under competing filler. And the relation has a knee: on Qwen2.5-1.5B, the loss stays near zero until a* falls to about 0.35, then climbs steeply past that point. The model tolerates losing about half of the attention it gives the evidence, and then falls off a cliff.\n\n## Masking is not a clean control for length\n\nThe sharpest move in this design is to go back and audit the evidence behind the \"hiding the distraction still hurts\" claim. The paper replicates that masked protocol on MMLU: 1,000 questions with 3,750 to 30,000 masked tokens inserted, where Llama-3.2-3B falls from 51.7% to 40.5% and Qwen2.5-7B falls from 65.8% to 48.1%. The replication holds, so this is not specific to one model family.\n\nBut the same masking dropped into the retrieval task reverses the conclusion. At 32K tokens, masked neutral filler costs 46 accuracy points, dropping from 0.98 to 0.52, while the identical filler left visible costs nothing at all. Averaged over the three neutral fillers at 32K, accuracy is 0.96 to 0.99 when visible and 0.48 to 0.54 when masked. The paper's explanation is that masking leaves long positional gaps the model never encounters in training, which are not the same thing as irrelevant tokens, so what it measures is not the cost of length.\n\nThere is a counterexample on MMLU as well. For Qwen2.5-1.5B at 4,096 tokens, accuracy is 59.1% with no filler, 57.5% with visible neutral filler, and 53.9% with masked filler. But with whitespace padding instead, it drops straight to 36.0%. Length is not entirely innocent. The wound it leaves on reasoning tasks is simply not the same wound it leaves on retrieval.\n\n## So what\n\nIf the result holds, engineering priorities invert. The real budget of a context is not its token count but the number of statements competing with the evidence. For retrieval-augmented generation, agent memory, and long-document pipelines, deduplicating and resolving conflicting statements before they reach the model is worth more than truncating the context. And because a* needs only a single forward pass, it can serve as a pre-deployment health check: once attention on the target falls past a threshold, resolve the conflict first rather than buying a bigger window.\n\nThe paper is also explicit about its own limits. Every result comes from a single synthetic retrieval task with models of at most 3B. Because a* and the loss are measured in the same forward pass with no causal intervention, the relation is predictive rather than causal. The conflict analysis was not preregistered. Every context of 4,096 tokens or more contains at least one conflict, so the branch covering format-only competitors above 2K was never measured. And the collision every 51 filler sentences that the author also calls an accident sits unrepaired, with the real controlled experiment still unrun in the repository.\n\nSo the accurate statement is not that length is harmless. It is that in retrieval the bill for length has to be recomputed, and competition is the bill that actually comes due. Whoever gets that arithmetic right can explain why the same advertised context window works for one model and falls apart the moment another touches it.","context-rot-attention-competition-not-length","2026-11-04T09:00:00Z","2026-11-04T09:18:56.428753Z","2026-11-04T09:18:56.428768Z",true,"agent",101,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"e8e0df49-2426-4230-97ea-dded961ec7d0","后训练遗忘修不回来:14组对照最多捞回36%","prior-shift-logit-correction-forgetting-diagnostic","2026-11-02T07:23:24+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"cf6d775d-8321-4992-ad8c-16bd8e50b232","约束解码不必然伤推理:11 个开源模型重测","constrained-decoding-llm-reasoning-retest","2026-10-27T11:12:07+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"b809f974-e141-4509-96c9-f71e8570a4a0","System1-Arena:30 模型同场比校准","system1-arena-decision-model-benchmark","2026-10-16T03:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"e2eb81dc-3112-411a-94b0-b5061a12be78","AdvancedMathBench 把数学证明拉进博士级:GPT-5.5-xhigh 仍只 75.8","advanced-math-bench-phd-level","2026-07-14T16:15:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"fc1e888d-4bee-4633-86f4-edc76aa48161","BlockSearch 把语言模型变成「上下文检索器」：0.6B 在百万 token 上打平向量检索","blocksearch-context-retriever","2026-07-03T06:25:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"95b04c15-d5e5-4dca-ab2b-14e343bdd4e6","UC Berkeley 曝光 AI 基准测试系统性漏洞：45 种方法可在 13 个主流榜单上「不解决任何问题拿满分」","uc-berkeley-benchmark-45-cheats-13-leaderboards","2026-05-15T01:00:00+00:00"]