[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-mllm-retrieval-heads-cross-modal-optimization":3,"topics-all":38,"news-related-58aafc2c-b297-4f8c-a817-cad6ed7f1eda":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"58aafc2c-b297-4f8c-a817-cad6ed7f1eda","跨模态检索头:18个模型实测,砍掉后音频检索归零","中科院信工所与腾讯团队在18个模型上普查多模态检索头:同一个答案用语音念出来或显示在屏幕上,动用的是同一撮头;跨模态共享的这部分不到全部注意力头的2%,砍掉后音频检索直接归零;再对这撮头的注意力做排序训练,文档检索立刻涨了六个点。","多模态大模型在长输入里找关键证据,靠的可能只是一小撮注意力头。中科院信工所与腾讯的团队 10 月 23 日放出的论文([arXiv:2610.32411](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.32411)),在 18 个模型 checkpoint 上系统摸底了这类「检索头」:音频、视觉这类新模态下它们依旧稀疏、大量跨模态共用,砍掉后音频检索直接归零;论文还把它们变成训练目标,让文档检索涨了约 6 个点。\n\n## 实验设计:同一个答案,念出来或亮出来\n\n检索头(retrieval heads)的概念来自文本模型研究:一小撮注意力头负责把上下文里的相关信息搬到输出端。这篇论文沿用一套校准打分——计算问题 token 对证据 token 的注意力,再用「无信息问题」在同一上下文下做基线扣除,差值排名前 5% 的头称为检索头。为了测新模态,团队构造了配对输入 AV-NIAH:往 MLVU 数据集的长视频里塞一个六位数「秘密数字」,音频条件下用合成语音念出来,视觉条件下在屏幕中央横幅显示一秒;背景视频、答案、插入深度、问题全部固定,变的只有证据模态。18 个 checkpoint 覆盖四组:纯文本(Qwen2.5-3B\u002F7B、Qwen3-4B\u002F8B)、图文(Qwen2.5-VL、Qwen3-VL 各两档、MiniCPM-V-4.5)、统一模型 BAGEL-7B,以及全能模型八款(Qwen2.5-Omni 两档、MiniCPM-o 两款、SALMONN 2+ 两档、SALMONN Pro 两档)。\n\n## 四个发现:稀疏、共用、保留、不可少\n\n**稀疏。**八个全能模型、两种证据模态下,不到十分之一的头就贡献了全部正分的一半,高分集中在中间层与中后层。\n\n**共用。**同一个数字念出来和显示出来,招揽的是同一批头:top-5% 检索头在音频\u002F视觉两个条件下重合 69%-83%,全排名 Spearman 相关 0.88-0.92,而随机集合的重合不到 3 个头。头的招募更多取决于「找什么」,而不是证据以什么模态出现。\n\n**保留。**模态扩张基本不动旧家底:SALMONN 系列与其图文底座的检索头相关系数 ≥0.97,Qwen2.5-Omni 也紧贴它出发的语言底座。\n\n**不可少。**把 top-5% 检索头消融掉,四个受测模型的音频检索全部归零——Qwen2.5-Omni-7B 的音频精确匹配从 98.3% 掉到 0,同规模的随机消融只把它降到 96.9%。更值得注意的是跨模态共享的那一小撮(Qwen2.5-Omni-7B 里 14 个头、SALMONN Pro 4B 里 10 个,不到全部头的 2%):砍掉它们,16 个模型-任务组合里 15 个的损伤超过砍图像检索头,且损伤集中在视觉数学与空间感知——MathVista 掉 22.3 个百分点,CV-Bench 掉 16.3,SALMONN Pro 4B 的视觉数学 MathVision 掉 18.8。\n\n## 从分析到优化:检索头当训练目标\n\n看懂机制之后,论文把检索头变成优化对象:固定选出的 16 个检索头,只对它们的注意力分数做排序训练——每个样本一页正例加一页同文档难负例,LoRA 挂在注意力投影上。在多模态文档检索基准 MMDocIR 上,Qwen3-VL-8B 的页面级 Micro Recall@1 从随机头的 38.00、未训练检索头的 56.45,训练后涨到 62.48,提升 6.03 个点,十个文档域全部提升;训练数据不含任何版面级标注,版面级检索仍从 47.14 涨到 48.60。换家族同样成立:Qwen2.5-VL-7B 涨 5.69 个点,Gemma3-12B 涨 2.17。四个域外基准的变化都在一个点以内。\n\n## 作者自己按了刹车\n\n62.48 的页面检索分其实超过了表中的已发表检索器(ColPali 54.58、Col-Phi3 57.10),但作者明确写了这些系统的评测设置不同,受控比较对象是同一设置下未训练的检索头;稿件标注 work in progress,所有结论都是论文自评,还没有第三方复现。\n\n## 所以呢\n\n这篇工作的价值在闭环:可解释性不止能「看」,还能直接变成训练信号——不到 2% 的头既解释了跨模态检索,也指出了该拧哪颗螺丝。对做长上下文、RAG 与多模态检索的工程团队,这是「先定位再训练」的一条可复制路径;下次看到全能模型找证据又快又准,可以想到背后可能就是那几十个头在干活。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.32411","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"ceb0b92c-9f41-4f2b-abd1-600e2c1f9b7b","en","MLLM Retrieval Heads: Sparse, Cross-Modal, and Trainable","Retrieval heads are shared across audio and visual evidence in 18 checkpoints; ablating them zeroes audio retrieval, training them adds 6.03 R@1.","Multimodal LLMs locating key evidence in long inputs may rely on a small cluster of attention heads. A team from the Institute of Information Engineering of the Chinese Academy of Sciences and Tencent, whose paper appeared on arXiv on October 23 ([arXiv:2610.32411](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.32411)), surveyed these \"retrieval heads\" across 18 model checkpoints: they remain sparse in new audio-visual modalities, are largely shared across modalities, and zero out audio retrieval when ablated — and the paper turns them into a training target that lifts document retrieval by about 6 points.\n\n## Design: the same answer, spoken or displayed\n\nThe notion of retrieval heads comes from language-model research: a sparse set of attention heads transfers relevant information from the context to the output. This paper reuses a calibrated scoring scheme — attention from question tokens to evidence tokens, minus a baseline computed with an uninformative question on the same context — and calls the top 5% of heads in that ranking retrieval heads. To probe new modalities, the authors built paired AV-NIAH inputs: a six-digit \"secret number\" is inserted into a long video from the MLVU dataset, either read by synthesized speech (audio condition) or shown in a centered banner for one second (visual condition). The source video, the answer, the insertion depth, and the question all stay fixed; only the evidence modality changes. The 18 checkpoints span four groups: text-only (Qwen2.5-3B\u002F7B, Qwen3-4B\u002F8B), vision-language (two sizes each of Qwen2.5-VL and Qwen3-VL, plus MiniCPM-V-4.5), the unified BAGEL-7B, and eight omni-modal models (two Qwen2.5-Omni sizes, two MiniCPM-o variants, two SALMONN 2+ sizes, and two SALMONN Pro sizes).\n\n## Four findings: sparse, shared, retained, necessary\n\n**Sparse.** Across all eight omni-modal models and both evidence modalities, fewer than one head in ten accounts for half of the total positive score, with high scores concentrated in middle and later-middle layers.\n\n**Shared.** The same number, spoken or displayed, recruits the same heads: top-5% retrieval heads overlap by 69%-83% between the audio and visual conditions, Spearman correlation over the full ranking sits between 0.88 and 0.92, while independent random sets would share fewer than three heads. Which heads fire depends more on what is being retrieved than on the modality carrying the evidence.\n\n**Retained.** Modality expansion barely touches the old machinery: SALMONN models correlate with their vision-language backbones at 0.97 or above, and Qwen2.5-Omni stays close to the language model it was adapted from.\n\n**Necessary.** Ablating the top-5% retrieval heads eliminates audio retrieval in all four tested models — in Qwen2.5-Omni-7B, audio exact match falls from 98.3% to 0, while matched random ablation leaves it at 96.9%. The small cross-modal subset is even more striking (14 heads in Qwen2.5-Omni-7B and 10 in SALMONN Pro 4B, fewer than 2% of all heads): removing it causes more damage than removing image-retrieval heads in 15 of 16 model-task pairs, with the damage concentrating on visual mathematics and spatial perception — MathVista drops 22.3 points, CV-Bench 16.3, and MathVision 18.8 on SALMONN Pro 4B.\n\n## From analysis to optimization: train the heads\n\nThe paper then treats retrieval-head attention as an optimization target: fix a set of 16 retrieval heads, and train only their attention scores with a ranking objective — one relevant page plus one hard negative from the same document per example, with LoRA adapters on the attention projections. On the multimodal document retrieval benchmark MMDocIR, the page-level Micro Recall@1 of Qwen3-VL-8B moves from 38.00 with random heads and 56.45 with untrained retrieval heads to 62.48 after ranking training, a gain of 6.03 points, holding in every one of the ten document domains; although the training data contain no layout-level supervision, layout retrieval still rises from 47.14 to 48.60. The effect carries across families: Qwen2.5-VL-7B gains 5.69 points and Gemma3-12B gains 2.17. Scores on four out-of-domain benchmarks change by less than one point.\n\n## The authors hit their own brakes\n\nThe 62.48 page score actually exceeds the published retrievers listed in the same table (ColPali at 54.58, Col-Phi3 at 57.10), but the authors explicitly note that evaluation settings differ, and treat the gain over untrained retrieval heads under identical settings as the controlled comparison. The manuscript is marked work in progress, every number is self-reported by the paper, and no third-party replication exists yet.\n\n## So what\n\nThe value here is the closed loop: interpretability is not just for looking — it can become a training signal. Fewer than 2% of the heads both explain cross-modal retrieval and point at which screw to turn. For teams working on long-context, RAG, or multimodal retrieval, this is a replicable \"locate first, train second\" recipe; the next time an omni-modal model finds evidence quickly and accurately, it may well be a few dozen heads doing the work.","mllm-retrieval-heads-cross-modal-optimization","2026-11-01T17:09:14Z","2026-11-01T17:09:16.749932Z","2026-11-01T17:09:16.749943Z",true,"agent",15,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"8a550513-ef18-4d19-b609-d5fee5a853af","LeCun证明柏拉图表示定理:109M JEPA对齐胜72B","lecun-platonic-representation-theorem-jepa","2026-10-30T23:11:14+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"eb55e1cf-15c1-487f-845c-e81f87f11cfa","红白机 CPU 编进 Qwen 权重:没训练,纯编译","nesformer-qwen-nes-cpu-weights","2026-10-31T17:30:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"e4ac848d-d24a-465f-bece-37e090da86de","全双工语音模型长对话必崩:KAIST 揪出初始化锚点被挤走","full-duplex-speech-collapse-fd-lhb-kv-cache","2026-10-27T03:20:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"a491d4db-2d5a-4723-b093-28d1512fd647","VLA-Δ:物理状态变化当监督信号,LIBERO 98.4","vla-delta-physical-state-transition","2026-10-20T19:07:27+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"fc533af5-9e8a-43c2-8408-2f931a6aa398","EngramEdit:把事实塞进 LLM 的记忆抽屉","engramedit-decoupled-knowledge-llm","2026-10-09T04:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"d78ea8a8-bebb-4b94-b340-127eb2874a73","Ovis 全模态嵌入 3B:综合分领先 5.19","ovis-omni-embedding-3b","2026-09-23T21:08:45+00:00"]