[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-slam-asr-induction-heads-noise-scaling":3,"topics-all":38,"news-related-72e668b2-3720-480b-af8d-9d14280bdf64":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"72e668b2-3720-480b-af8d-9d14280bdf64","3%的注意力头,撑起冻结LLM的语音识别","SLAM-ASR只训一个小投影层,语音编码器和LLM全部冻结,就能做出有竞争力的语音识别。爱沙尼亚团队找到了电路层面的原因:真正干活的是冻结LLM里的感应头,挡掉前10个词错误率直接翻倍;推理时把这3%的头放大1.5倍,还能在强噪声下白捡几个点的降噪,Gemma除外。","SLAM-ASR 是语音识别里一条出了名省事的路线: 语音编码器(Whisper turbo)冻结, 大语言模型也冻结, 只训一个小投影层——把每 5 个连续编码器状态拼起来, 过一个隐藏层 2048 的两层 MLP——把两边接起来, 转写效果就有竞争力; 原始论文甚至把方法自嘲为 embarrassingly simple。但一个疑问始终没人回答: 一个小映射, 凭什么够用? 爱沙尼亚塔林理工大学(Tallinn University of Technology)的 Fedorchenko 与 Alumäe 两位研究者给出电路层面的答案(https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.24721): 真正干活的, 是冻结 LLM 里的感应头。\n\n## 感应头: LLM 自带的复读电路\n\n感应头(induction head)是文本 LLM 里被反复研究的电路: 看到 A 后面跟过 B, 下次再遇到 A, 它就注意 B 并抬高 B 的输出概率, 一边做前缀匹配, 一边做复制。Anthropic 的 Olsson 等人 2022 年在 Transformer Circuits 系列里系统描述过它(https:\u002F\u002Ftransformer-circuits.pub\u002F2022\u002Fin-context-learning-and-induction-heads\u002Findex.html), 独立团队后来也在 NAACL 2025 的工作里把它确认为上下文学习做模式匹配的关键机制。这篇论文找头的方式很干净: 完全不看语音, 用 50 个随机词重复两遍的序列算每个头的前缀匹配得分, 取排名前 3% 的集合 Ck——Llama 3.2-3B 里 20 个头, Gemma 2-2B 里 6 个, Ministral 3-3B 里 25 个。\n\n## 挡住 10 个头, 错误率翻倍\n\n因果验证靠注意力屏蔽: 把这些头对语音位置的注意力权重压到负无穷, 让注意力重新分配到其余上下文。Llama 3.2-3B 上, 挡掉前 10 个感应头, 词错误率(WER)从 3.3-3.6 涨到 7.1-8.3, 直接翻倍; 从同样层里随机挑 10 个头挡掉, 错误率几乎不动。三个解码器上挡 C10 都会推高 WER。论文的解释是, 语音嵌入在这里扮演序列里第一次出现的角色, 每个词的语音嵌入近似它生成的 token, 于是站在刚生成的词上, 感应头照旧注意语音里排在后面的那个词, 把它抄出来。\n\n## 成对头: 模型自己长出的差分注意力\n\n更有意思的是成对结构。作者把每层里对 ASR 最重要的头记为 h1, 再找同层输出方向与它最相反的头 h2, 发现有的对在分组查询注意力(GQA)下共享同一组键值: 两边注意到的位置一样, 输出却反向, 叠加后共同的注意力噪声被抵消, 注意力被锐化到刚生成的那个词上。这正是 ICLR 2025 差分 Transformer 的设计思路, 区别是它不是架构设计出来的, 是模型自己长出来的。挡 h1 一个头比把一对都挡掉伤害大得多; 单挡 h2 在 Gemma 和 Ministral 上几乎没影响。\n\n## 顺手捡到一个降噪开关\n\n反过来加强这批头会怎样? 论文在推理时把 Ck 的输出放大 1.5 倍, 在 400 条 test-clean 加 400 条 test-other 语音上加气泡声和白噪声(信噪比 5 和 0 dB)测试。Llama 在最难的气泡声 0 dB 条件下, WER 从 89.0 降到 81.8, 插入率从 40.4 降到 32.1; 加噪均值 WER 从 41.9 到 39.9。对照里, 放大同层其余的头, 加噪均值 WER 冲到 50.8, 比不放大还差; 三次随机种子下 Ck 都稳定优于随机头。干净音频上的代价最多 0.2 个点。此前 ICASSP Workshops 2025 的独立评估已经发现 SLAM-ASR 在加性噪声下退化严重, 这个开关算一个便宜补丁。不过降噪只在 3 个 LLM 里的 2 个上成立, Gemma 上并无效。\n\n## 该泼的冷水\n\n所有数值都是论文自报; 训练只用 LibriSpeech 的 clean-100 子集, 噪声测试规模是 800 条语音; 「投影层该怎么训, 要不要连着头一起更新」在论文里仍是开放问题——按论文的描述, 现在的投影层只是把语音糊成一种带噪声的重复文本, 让 LLM 的旧电路勉强能读。\n\n所以呢: 这篇论文的价值不在刷榜, 而是把「冻结复用」路线的账算清了——你不是在训一个语音识别器, 你是在给 LLM 已有的电路接线。机制解释直接变现成推理时的干预手段, 一个放大系数就白捡几个点的降噪。下次看到「只训百分之一参数对齐多模态」的工作, 不妨先问一句: 冻结的那头, 到底是哪些头在干活?","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.24721","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"c4455801-0273-477d-9e92-00b4addfa439","asr",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"1ccf2944-ebbe-4c78-834a-73d96edc89a9","en","3% of Attention Heads Carry Frozen-LLM Speech Recognition","Induction heads carry frozen-LLM ASR: blocking the top 10 doubles WER; scaling them 1.5x at inference trims noisy errors for free, except on Gemma.","SLAM-ASR is a famously lazy recipe for speech recognition: freeze the speech encoder (Whisper turbo), freeze the large language model, and train only a small projector — a two-layer MLP with hidden size 2048 that concatenates every 5 consecutive encoder states — to wire the two together. The original paper literally calls the approach embarrassingly simple, and it still yields competitive transcription. One question, however, stayed unanswered: why is a small trained map enough? Artem Fedorchenko and Tanel Alumäe of Tallinn University of Technology, Estonia, now give a circuit-level answer (https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.24721): the components doing the real work are the induction heads inside the frozen LLM.\n\n## Induction heads: the copying circuit LLMs already have\n\nAn induction head is one of the best-studied circuits in text LLMs: if it has seen A followed by B, then on the next occurrence of A it attends to B and raises the logit of B — prefix matching plus copying, in one head. Anthropic's Olsson et al. described the mechanism systematically in the 2022 Transformer Circuits series (https:\u002F\u002Ftransformer-circuits.pub\u002F2022\u002Fin-context-learning-and-induction-heads\u002Findex.html), and an independent team later confirmed induction heads as an essential mechanism for pattern matching in in-context learning at NAACL 2025. What is clean about the new paper is how it finds the heads: it never looks at speech. It feeds each model 40 sequences of 50 random vocabulary tokens repeated twice, computes a prefix-matching score per head, and keeps the top 3 percent as the set Ck — 20 heads in Llama 3.2-3B, 6 in Gemma 2-2B, and 25 in Ministral 3-3B.\n\n## Block ten heads and the error rate doubles\n\nThe causal test is attention blocking: the selected heads' attention logits on speech positions are set to negative infinity, forcing each head's attention to redistribute over the rest of the context. On Llama 3.2-3B, blocking the top 10 induction heads pushes word error rate (WER) from 3.3-3.6 up to 7.1-8.3 — roughly doubled — while blocking 10 random heads drawn from the same layers leaves WER essentially unchanged. Blocking the top-10 set raises WER in all three decoders. The paper's reading: speech embeddings play the role of the first occurrence in the sequence. Each word's speech embeddings approximate its generated token, so from the just-generated word, an induction head attends to the speech embeddings of the word that comes next and copies it out — the same old circuit, rerouted through audio.\n\n## Pairs of heads: differential attention that grew on its own\n\nThe more striking find is a pairing structure. In each layer the authors take h1, the head with the largest ASR importance, and h2, the head of the same layer whose output direction is most anti-aligned with h1. Some of these pairs share a single key\u002Fvalue group under grouped-query attention: both heads attend to the same positions, but their outputs point in opposite directions, so the shared attention noise cancels and attention sharpens onto the previously generated word. That is precisely the idea of the Differential Transformer from ICLR 2025 — except here nobody designed it; the model grew it. Blocking h1 alone hurts far more than blocking both heads together, while blocking h2 alone leaves WER unchanged in Gemma and Ministral.\n\n## A noise-reduction switch, picked up for free\n\nWhat happens if you amplify these heads instead? At inference time the paper scales the outputs of Ck by 1.5, and evaluates on 400 test-clean plus 400 test-other utterances with babble and white noise added at 5 and 0 dB SNR. On Llama, in the hardest condition — babble at 0 dB — WER drops from 89.0 to 81.8 and the insertion rate from 40.4 to 32.1; the mean WER under noise goes from 41.9 to 39.9. In the control, scaling the remaining heads of the same layers pushes noisy-mean WER up to 50.8, worse than not scaling at all, and across 3 random seeds Ck scaling is consistently better than scaling random heads. The cost on clean audio is at most 0.2 points. An independent evaluation at ICASSP Workshops 2025 had already found that SLAM-ASR degrades significantly under additive noise, which makes this a cheap patch. Note, though, that the trick works in only 2 of the 3 LLMs — it does not help on Gemma.\n\n## The cold water\n\nEvery number here is self-reported by the paper; training used only the clean-100 subset of LibriSpeech, and the noise evaluation covers 800 utterances. How the projector should be trained, and whether the heads should be updated alongside it, remains an open question in the paper — on its own account, today's projector merely smears speech into a noisy and redundant version of the transcript, so the LLM's old circuits can just about read it.\n\nSo what? The value of this paper is not another benchmark point. It is an honest audit of the freeze-and-reuse route: you are not training a speech recognizer, you are wiring up circuits the LLM already has. The mechanistic explanation converts directly into an inference-time intervention — one scaling factor buys a few points of noise robustness for free. The next time you see a paper that aligns a modality by training one percent of the parameters, ask the obvious question: on the frozen side, which heads are actually doing the work?","slam-asr-induction-heads-noise-scaling","2026-11-19T21:15:00Z","2026-11-19T21:10:09.133441Z","2026-11-19T21:10:09.133456Z",true,"agent",56,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"58aafc2c-b297-4f8c-a817-cad6ed7f1eda","跨模态检索头:18个模型实测,砍掉后音频检索归零","mllm-retrieval-heads-cross-modal-optimization","2026-11-01T17:09:14+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"8a550513-ef18-4d19-b609-d5fee5a853af","LeCun证明柏拉图表示定理:109M JEPA对齐胜72B","lecun-platonic-representation-theorem-jepa","2026-10-30T23:11:14+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"f5c0b227-faf9-47e3-863f-3c102365cd41","LongCat-Next 开源：把文字、图像和声音统一成离散 Token","longcat-next-discrete-native-multimodal","2026-08-09T08:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"a151db0c-d832-4df2-ac03-2d4e58b26e99","Kimi K3 跑通 MiniTriton:Moonshot 让 LLM 第一次从零编译出自己的 GPU 编译器","kimi-k3-minitriton-gpu-compiler","2026-07-26T14:00:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"386cd824-56cc-4474-8388-8caef2dbb9dd","Rosetta：让多模态预训练不再遗忘——腾讯混元与港科大提出 MAOP 零开销投影法","tencent-rosetta-maop","2026-07-03T00:02:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"872f163b-5ed4-4405-91aa-48c0fe34c25f","论文实测:模型在哪一层想通这个词,能预测人读它多久","resolution-depth-word-reading-time-llm","2026-11-22T05:00:00+00:00"]