[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-alibi-attention-underflow-long-context":3,"news-related-0a3f5044-ed20-4c2a-b710-bd26cd276d3e":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"0a3f5044-ed20-4c2a-b710-bd26cd276d3e","ALiBi 的隐藏数值故障：长上下文越长，部分注意力头越可能“失明”","一篇 8 月 4 日提交的论文指出，ALiBi 位置编码的线性距离偏置会在有限浮点精度下把部分 softmax 权重压到零，使注意力头无法读取超出特定距离的 token。研究者在预训练模型和 1.48 亿参数解码器实验中复现了该问题，并比较了截断、斜率重设、对数距离和 soft capping 四类缓解方案。","# ALiBi 的隐藏数值故障：长上下文越长，部分注意力头越可能“失明”\n\nALiBi 常被视为一种便宜、无额外参数、适合做长度外推的位置编码。但一篇 8 月 4 日提交到 arXiv 的论文指出，它有一个此前容易被忽略的数值故障：**上下文距离增大时，线性偏置可能把 softmax 中的指数压到浮点格式无法表示的范围，最终让对应注意力权重直接变成零。**研究者把这种现象称为注意力头的“部分失明”。[论文原文](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.03994)\n\n## 问题不在公式，而在有限精度\n\nALiBi 会根据 token 之间的距离，给注意力 logits 加上按注意力头区分的负偏置。距离越远，偏置越负。在数学表达里，权重可以无限接近零；但在 fp32 或 bf16 计算中，指数一旦低过格式能表示的最小正数，就会发生下溢，权重不是“很小”，而是精确等于零。\n\n论文给出的简化分析显示，fp32 和 bf16 的下溢阈值分别约为 -103.27 与 -92.18。默认 ALiBi 为不同注意力头设置不同斜率，因此斜率更陡的头会更早跨过阈值。以 16 个注意力头和 bf16 为例，当 token 距离达到 2048 时，论文估算有 36.6% 的注意力矩阵条目已跨过下溢阈值。\n\n这会带来两层后果。第一，超出“失明距离”的 token 权重为零，不能再向当前 token 传递信息。第二，这些归零项也会退出 softmax 分母，剩余 token 的权重会被重新分配。也就是说，它不只是弱化远距离信息，而是悄悄改变了整个注意力分布。\n\n## 常规榜单可能看不出来\n\n研究者检查了 BLOOM、Falcon-RW 和 MPT 等采用 ALiBi 的预训练模型，并用 1.48 亿参数的解码器进行控制实验。结果很值得警惕：这种故障会明显伤害 token 检索，但在常见解码器 benchmark 上影响较小。论文结论部分称，常规下游任务的差异只有 1.6 到 3.4 个百分点，而 passkey 与 needle-in-a-haystack 检索探针显示出更大的变化。\n\n这解释了为什么问题可能长期不显眼：模型在一般问答、常识和语言任务上仍能交出看似正常的成绩，但需要跨长距离精确取回信息时，部分注意力容量已经消失。论文也没有把 ALiBi 判为不可用；相反，默认斜率在部分检索测试中仍是很强的基线，甚至没有一种缓解方案能同时赢下所有任务。\n\n## 四种修补方式，没有万能答案\n\n论文比较了四类训练期策略：\n\n- **截断偏置**：给负偏置设置下限，避免继续跌入下溢区间；\n- **重设斜率**：让各注意力头的失明距离覆盖目标上下文范围；\n- **对数化距离**：用距离的对数替代线性距离，大幅推迟下溢；\n- **soft capping**：限制注意力 logits 的幅度，减少极端值把指数推过阈值。\n\n在 passkey 的上下文外检索中，“截断 + 对数距离”组合的 AUC 从 ALiBi 基线的 0.08 提升到 0.79，接近十倍；但在另一项 needle-in-a-haystack 测试上，默认 ALiBi 仍表现最好。研究者因此没有宣布单一赢家，只建议：如果目标本来就是软滑动窗口，截断应成为默认选择；如果采用硬滑动窗口，则可以直接跳过窗口外计算，而不是先算出权重再让它下溢为零。\n\n## 真正的提醒：长上下文不能只看标称长度\n\n这项研究最重要的价值，不是证明某个位置编码“坏了”，而是把长上下文评估从架构承诺拉回数值现实。模型支持多少 token，只说明输入能被送进去；远处信息是否仍有非零权重、梯度能否穿过、检索能力是否保持，是另一套问题。\n\n论文也明确给出边界：主要训练实验使用的是 1.48 亿参数模型，部分观察未必能直接外推到更大的模型。可解析的下溢机制成立，但不同架构、语料和训练规模下的行为仍需继续验证。\n\n所以，下一次看到“支持更长上下文”，更该追问的不是窗口有多大，而是：**窗口最远处的信息，模型究竟还能不能真正看见？**","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.03994","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"34d6094a-0bde-4fe7-ab7d-7718083dda2b","en","ALiBi's hidden numeric fault blinds attention heads","A paper submitted on August 4 identifies a numerical failure in ALiBi positional encoding: its linear distance bias can push softmax weights below floating-point precision, turning them into exact zeros and preventing attention heads from reading sufficiently distant tokens. The authors reproduce the effect in pretrained models and 148M-parameter decoder experiments, then evaluate four mitigation families.","# ALiBi's Hidden Numerical Failure: Longer Contexts Can Make Attention Heads Go Blind\n\nALiBi is often treated as a cheap, parameter-free positional encoding that supports length extrapolation. A paper submitted to arXiv on August 4 identifies an easily overlooked numerical failure: **as context distance grows, ALiBi's linear bias can push softmax exponentials outside the range representable by floating-point formats, turning some attention weights into exact zeros.** The authors describe the affected heads as becoming partially blind. [Read the paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.03994).\n\n## The formula is not the problem; finite precision is\n\nALiBi adds a head-specific negative bias to attention logits according to the distance between tokens. The farther apart two tokens are, the more negative the bias becomes. In exact mathematics, the resulting weight can approach zero indefinitely. In fp32 or bf16 arithmetic, however, an exponential below the smallest representable positive number underflows. The weight is no longer merely tiny; it becomes exactly zero.\n\nThe paper's simplified analysis places the underflow thresholds for fp32 and bf16 at approximately -103.27 and -92.18, respectively. Because default ALiBi assigns different slopes to different attention heads, heads with steeper slopes cross the threshold sooner. For a 16-head layer using bf16, the authors estimate that at a token distance of 2,048, 36.6% of the attention-matrix entries have crossed the underflow threshold.\n\nThe failure has two consequences. First, tokens beyond a head's blindness distance receive zero weight and cannot contribute information to the current token. Second, those zeroed terms also disappear from the softmax denominator, redistributing probability mass over the remaining tokens. The mechanism therefore does more than weaken distant information: it silently changes the full attention distribution.\n\n## Standard benchmarks may barely notice\n\nThe researchers inspect pretrained ALiBi models including BLOOM, Falcon-RW, and MPT, and run controlled pretraining experiments with 148M-parameter decoder models. The result is uncomfortable: the failure can substantially damage token retrieval while producing only small changes on standard decoder benchmarks. The paper reports differences of 1.6 to 3.4 percentage points on regular downstream tasks, while passkey and needle-in-a-haystack retrieval probes reveal much larger effects.\n\nThat gap helps explain why the issue may have remained obscure. A model can still look normal on general question answering, commonsense, and language tasks even after part of its long-range attention capacity has vanished. The paper does not conclude that ALiBi is unusable. Default slopes remain a strong baseline on some retrieval tests, and no mitigation wins every task.\n\n## Four mitigation families, but no universal fix\n\nThe paper evaluates four training-time strategies:\n\n- **Bias clamping:** impose a lower bound on the negative bias so it cannot enter the underflow region;\n- **Robust slopes:** choose head slopes so their blindness distances cover the intended context range;\n- **Log-scaled distance:** replace linear distance with its logarithm, delaying underflow dramatically;\n- **Soft capping:** bound the magnitude of attention logits so extreme values are less likely to push exponentials beyond the threshold.\n\nFor out-of-context passkey retrieval, the combination of clamping and log-scaled distance raises AUC from the ALiBi baseline of 0.08 to 0.79, nearly a tenfold increase. Yet default ALiBi still performs best on the paper's other needle-in-a-haystack test. The authors therefore avoid declaring a single winner. They recommend clamping when the desired behavior is already a soft sliding window. If a hard sliding window is acceptable, computation outside the window can be skipped explicitly instead of being performed and then discarded through underflow.\n\n## The larger lesson: advertised context length is not usable context\n\nThe paper's main value is not a verdict that one positional encoding is broken. It pulls long-context evaluation away from architectural promises and back toward numerical behavior. A model accepting a given number of tokens only proves that the input fits. Whether distant information retains nonzero weight, whether gradients still pass through it, and whether retrieval survives are separate questions.\n\nThe authors also state an important limitation: most controlled training experiments use 148M-parameter models, so every empirical observation may not transfer directly to larger systems. The analytical underflow mechanism is grounded, but interactions with other architectures, corpora, and training scales still require study.\n\nThe next time a model advertises a longer context window, the sharper question is not how many tokens fit. It is this: **at the far end of that window, can the model still truly see the information?**","alibi-attention-underflow-long-context","2026-08-06T10:30:00Z","2026-08-05T18:15:22.708570Z","2026-08-05T18:15:22.708578Z",true,"agent",89,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"f9cf9f03-6aca-4d29-94d3-5c6acfeaf435","匿名模型 OX Alpha 短暂登顶 OpenRouter 编码榜:研究者推测底座指向智谱 GLM-5.x","ox-alpha-stealth-openrouter-glm-5-zhipu","2026-08-24T03:00:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"43eda321-b0b7-4df7-b20e-9758cbab42c9","记忆越完整,眼前题越做不对:MemTrapBench 把 LLM 长期记忆框架打回原形","memtrapbench-llm-memory-cognitive-traps","2026-08-22T04:00:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"b05de01b-89ca-499b-b130-e55162e651f5","SCOPE：让大模型学会选择性信任，而不是把上下文一概拒绝","scope-selective-trust-context-dpo","2026-08-06T17:59:58+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"3c6fcf46-f5bb-4136-931c-69cd64216e12","Skill-Use 基准揭示 Agent 短板：会做任务，不等于会用 Skill","skill-use-agent-harness-benchmark","2026-08-06T08:00:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"74de194b-9e2c-45ab-aa13-12fe210e66ba","HiGram 给 Agent 记忆加上“路径定位”：先找证据，再改记忆","higram-agent-memory-path-localization","2026-08-05T09:32:43+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"e3f049f5-2f0d-48d2-8e88-246ef006fa16","LoopMTP 给循环 Transformer 装上前瞻路标：固定参数下让每一轮都做不同的事","loopmtp-latent-multi-token-loop-guidance","2026-08-04T13:13:09+00:00"]