[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-rho-invariant-predicts-induction-heads":3,"topics-all":38,"news-related-27b8351d-d0f7-4e89-b5bc-ad3a3fce9a50":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"27b8351d-d0f7-4e89-b5bc-ad3a3fce9a50","不跑模型不喂数据:一个标量预测归纳头","机制可解释性通常要跑激活、造探针，NeurIPS 2026 NeurReps 工作坊论文反其道而行：从注意力权重推出标量不变量 ρ，零前向零数据。四个 135M-7B RoPE 模型上，ρ 与归纳头得分相关达 -0.74，最强十个归纳头全在低 ρ 尾部；用 ρ 把 YaRN 只加到一半头，困惑度损失也小于全头缩放。","找出一个注意力头究竟在干什么，标准做法是跑数据：激活补丁、探针分类器、精心构造的输入，全都离不开前向传播。Miguel P. Bento 的 arXiv 2611.12206 给出另一条路：把注意力机制看作带连续对称性的几何对象，从权重里直接算出一个标量 ρ，一次前向都不跑、一条数据都不喂，就能判断每个头耦合位置信息的能力有多强。论文已被 NeurIPS 2026 的 Symmetry and Geometry in Neural Representations（NeurReps）工作坊接收为扩展摘要。\n\n## ρ：位置扰动相对静态项的比重\n\n在 RoPE 模型里，注意力 logit 可以写成静态内容项 G_QK(0)=W_Q^T W_K 加上位置扰动项 P(Δ)=W_Q^T[R(Δ)−I]W_K，R(Δ) 是相对位置 Δ 上的旋转。ρ 的定义很朴素：在一组对数间隔的 Δ 上，取 ||P(Δ)||²_F 与 ||G_QK(0)||²_F 的比值再平均——位置项相对内容项有多大，ρ 就有多大。\n\n这个数有两层性质。其一，它是基不变的：G_QK 在 (W_Q, W_K) 的整族重参数化下保持不动，ρ 继承了这份对称性，还额外对权重缩放免疫。其二，它度量的是容量而非用法：ρ=0 的头对任何输入都产生与位置无关的 logits（论文 Proposition 1 严格化了这一点），而高 ρ 只说明这个头有能力耦合位置，不代表它在某条输入上真的这么做。\n\n## 归纳头全躲在低 ρ 尾部\n\n归纳头实现「A B … A → B」的复制模式，是机制可解释性里被研究得最多的电路。它做的是内容匹配，天然与位置无关，于是作者的预测是：归纳头的 ρ 应该极低。四个无偏置 RoPE 模型给出整齐的验证：SmolLM2-135M、TinyLlama-1.1B、SmolLM2-1.7B、Mistral-7B，ρ 与归纳得分的 Spearman 相关系数从 -0.55 到 -0.74，全部 p\u003C10⁻⁴⁰；每个模型最强的十个归纳头，无一例外落在 ρ 分布最低的 7%-30% 尾部。\n\n作者也坦承相关性并不完美：归纳是多头电路，低 ρ 是必要条件而非充分条件——这个「不完美」恰好与「容量」解释自洽。\n\n## 用 ρ 门控 YaRN：只动一半的头\n\n均匀的 RoPE 缩放之所以伤短上下文，是因为它不加区分地修改所有头。作者反过来用 ρ 做门控：只对 ρ 最高的 50% 头应用 YaRN，其余头保持原始位置。在 Mistral-7B 上（原生上下文窗口内评测），512 token 短上下文困惑度：基线 7.63，全头 YaRN 19.37，定向 YaRN 12.92；4096 token 长上下文：基线 5.44，全头 162.96，定向 130.34。论文明确说这只是初步的概念验证。\n\n## 所以呢\n\n边界写得清楚：只验证了 RoPE 架构的 QK 通路、模型最大到 7B、YaRN 应用只在单个模型上做过。作者还预告了下一步：G_QK 特征谱、G_VO 奇异值、两者的交换子，可以拼出每个头的「不变量签名」。这个方向真正的卖点在成本结构——权重是静态、便宜、可复制的，激活分析又贵又依赖数据选择；如果不变量在更大模型上依然成立，可解释性就多了一个「先筛后查」的廉价前置步骤。代码开在 [github.com\u002FMPedraBento\u002Frho-invariant](https:\u002F\u002Fgithub.com\u002FMPedraBento\u002Frho-invariant)，论文见 [arxiv.org\u002Fabs\u002F2611.12206](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.12206)。当分析一个模型的成本降到读它的权重本身，你第一个想看的会是哪个头？","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.12206","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"9e367663-c873-4554-b9f0-00bfa195c757","en","One Scalar From Weights Alone Predicts Induction Heads","Scalar ρ from attention weights alone — no forward pass, no data — predicts induction heads (Spearman to -0.74) on four RoPE models, NeurReps 2026.","Finding out what an attention head actually does normally means running data: activation patching, probing classifiers, crafted inputs — all of it needs forward passes. An arXiv paper (2611.12206) by Miguel P. Bento takes the opposite route: treat attention as a geometric object with continuous symmetries, compute a single scalar ρ straight from the weights, and characterize heads without a single forward pass or token of data. The paper was accepted as an extended abstract at the Symmetry and Geometry in Neural Representations Workshop (NeurReps), NeurIPS 2026.\n\n## What ρ measures\n\nIn RoPE models the attention logit splits into a static content term G_QK(0) = W_Q^T W_K plus a positional perturbation P(Δ) = W_Q^T [R(Δ) − I] W_K, where R(Δ) is the rotation at relative position Δ. ρ is simply the average of ||P(Δ)||²_F over ||G_QK(0)||²_F across a logarithmically spaced set of offsets — how big the positional term is relative to the content term.\n\nTwo properties matter. First, ρ is basis-invariant: G_QK is unchanged under the whole family of (W_Q, W_K) reparameterizations, and ρ inherits that invariance plus scale invariance. Second, it measures capacity, not usage: a head with ρ = 0 produces position-independent logits for any input (Proposition 1), while a high-ρ head merely has the capacity to couple to position — it need not do so on a given input. One caveat the authors note: ρ is sensitive to softmax shifts, though weight decay tends to suppress the offending components in practice.\n\n## Induction heads hide in the low-ρ tail\n\nInduction heads implement the \"A B … A → B\" copying pattern, the most-studied circuit in mechanistic interpretability. Content matching is position-invariant by nature, so the prediction is that induction heads have extremely low ρ. Four bias-free RoPE models confirm it: SmolLM2-135M, TinyLlama-1.1B, SmolLM2-1.7B and Mistral-7B show Spearman correlations between ρ and induction scores from -0.55 to -0.74, all p \u003C 10⁻⁴⁰, and the ten strongest induction heads always sit in the bottom 7%-30% of the ρ distribution.\n\nThe correlation is imperfect, and the authors say why: induction is a multi-head circuit, so low ρ is necessary but not sufficient — which is exactly the capacity interpretation.\n\n## Gating YaRN with ρ\n\nUniform RoPE scaling hurts short-context performance because it modifies every head indiscriminately. The authors instead apply YaRN only to the top-50% of heads by ρ, leaving the rest at their original positions. On Mistral-7B, evaluated within the native context window: at 512 tokens, perplexity is 7.63 for the baseline, 19.37 for uniform YaRN and 12.92 for the targeted variant; at 4096 tokens, 5.44, 162.96 and 130.34 respectively. The authors explicitly call this a preliminary proof of concept.\n\n## So what\n\nThe boundaries are stated plainly: RoPE architectures only, the QK sector only, models up to 7B, and the YaRN experiment on a single model. The authors also sketch where this generalizes: the G_QK eigenspectrum, the singular values of G_VO, and the commutator between the two could form an invariant signature per head — a data-independent counterpart to activation-based circuit maps.\n\nThe interesting part is the cost structure. Weights are static, cheap and reproducible; activation analysis is expensive and data-dependent. If invariants keep holding at larger scales, interpretability gains a cheap \"screen first, inspect later\" step. Code is on GitHub (github.com\u002FMPedraBento\u002Frho-invariant); the paper is at arxiv.org\u002Fabs\u002F2611.12206. When reading a model's weights is enough to read its heads, which head would you check first?","rho-invariant-predicts-induction-heads","2026-11-10T21:20:00Z","2026-11-10T21:07:51.017916Z","2026-11-10T21:07:51.017927Z",true,"agent",41,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"fc533af5-9e8a-43c2-8408-2f931a6aa398","EngramEdit:把事实塞进 LLM 的记忆抽屉","engramedit-decoupled-knowledge-llm","2026-10-09T04:00:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"cb7fb8b3-5862-4cba-adab-c4794e989966","图灵奖得主 Pearl 长访谈：LLM 能讲因果只是因为人类替它爬过了因果阶梯","judah-pearl-llm-causal-ladder-agi","2026-07-31T07:00:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"f8436dd3-d6fc-4ea7-9f2e-1086026c11d0","Transformer 的几何之眼：arXiv 2607.17146 把注意力炼成薛定谔桥，把 SGD 写成伊藤扩散","transformer-geometry-schrodinger-bridge","2026-07-23T12:10:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"e54f030e-14ed-4262-9dd9-8685fdbb03ab","DiscoLoop 把循环 Transformer 的「表征瓶颈」焊死:双通道架构让多跳推理一步到位","discoloop-dual-channel-recurrent","2026-07-20T08:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"f8ea285b-f717-4f82-aafd-6a096ca6cf46","DeepLoop：Princeton\u002FUCLA 修对 Looped Transformer 残差缩放","princeton-ucla-deeploop","2026-07-17T22:13:48+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"78b52428-2dbb-4819-aa2a-eee5bccf7d5a","用大模型筛基准:基准反而变简单了","llm-validation-minimal-pair-benchmarks-easier","2026-11-11T11:20:00+00:00"]