Multimodal LLMs locating key evidence in long inputs may rely on a small cluster of attention heads. A team from the Institute of Information Engineering of the Chinese Academy of Sciences and Tencent, whose paper appeared on arXiv on October 23 (arXiv:2610.32411), surveyed these "retrieval heads" across 18 model checkpoints: they remain sparse in new audio-visual modalities, are largely shared across modalities, and zero out audio retrieval when ablated — and the paper turns them into a training target that lifts document retrieval by about 6 points.

Design: the same answer, spoken or displayed

The notion of retrieval heads comes from language-model research: a sparse set of attention heads transfers relevant information from the context to the output. This paper reuses a calibrated scoring scheme — attention from question tokens to evidence tokens, minus a baseline computed with an uninformative question on the same context — and calls the top 5% of heads in that ranking retrieval heads. To probe new modalities, the authors built paired AV-NIAH inputs: a six-digit "secret number" is inserted into a long video from the MLVU dataset, either read by synthesized speech (audio condition) or shown in a centered banner for one second (visual condition). The source video, the answer, the insertion depth, and the question all stay fixed; only the evidence modality changes. The 18 checkpoints span four groups: text-only (Qwen2.5-3B/7B, Qwen3-4B/8B), vision-language (two sizes each of Qwen2.5-VL and Qwen3-VL, plus MiniCPM-V-4.5), the unified BAGEL-7B, and eight omni-modal models (two Qwen2.5-Omni sizes, two MiniCPM-o variants, two SALMONN 2+ sizes, and two SALMONN Pro sizes).

Four findings: sparse, shared, retained, necessary

Sparse. Across all eight omni-modal models and both evidence modalities, fewer than one head in ten accounts for half of the total positive score, with high scores concentrated in middle and later-middle layers.

Shared. The same number, spoken or displayed, recruits the same heads: top-5% retrieval heads overlap by 69%-83% between the audio and visual conditions, Spearman correlation over the full ranking sits between 0.88 and 0.92, while independent random sets would share fewer than three heads. Which heads fire depends more on what is being retrieved than on the modality carrying the evidence.

Retained. Modality expansion barely touches the old machinery: SALMONN models correlate with their vision-language backbones at 0.97 or above, and Qwen2.5-Omni stays close to the language model it was adapted from.

Necessary. Ablating the top-5% retrieval heads eliminates audio retrieval in all four tested models — in Qwen2.5-Omni-7B, audio exact match falls from 98.3% to 0, while matched random ablation leaves it at 96.9%. The small cross-modal subset is even more striking (14 heads in Qwen2.5-Omni-7B and 10 in SALMONN Pro 4B, fewer than 2% of all heads): removing it causes more damage than removing image-retrieval heads in 15 of 16 model-task pairs, with the damage concentrating on visual mathematics and spatial perception — MathVista drops 22.3 points, CV-Bench 16.3, and MathVision 18.8 on SALMONN Pro 4B.

From analysis to optimization: train the heads

The paper then treats retrieval-head attention as an optimization target: fix a set of 16 retrieval heads, and train only their attention scores with a ranking objective — one relevant page plus one hard negative from the same document per example, with LoRA adapters on the attention projections. On the multimodal document retrieval benchmark MMDocIR, the page-level Micro Recall@1 of Qwen3-VL-8B moves from 38.00 with random heads and 56.45 with untrained retrieval heads to 62.48 after ranking training, a gain of 6.03 points, holding in every one of the ten document domains; although the training data contain no layout-level supervision, layout retrieval still rises from 47.14 to 48.60. The effect carries across families: Qwen2.5-VL-7B gains 5.69 points and Gemma3-12B gains 2.17. Scores on four out-of-domain benchmarks change by less than one point.

The authors hit their own brakes

The 62.48 page score actually exceeds the published retrievers listed in the same table (ColPali at 54.58, Col-Phi3 at 57.10), but the authors explicitly note that evaluation settings differ, and treat the gain over untrained retrieval heads under identical settings as the controlled comparison. The manuscript is marked work in progress, every number is self-reported by the paper, and no third-party replication exists yet.

So what

The value here is the closed loop: interpretability is not just for looking — it can become a training signal. Fewer than 2% of the heads both explain cross-modal retrieval and point at which screw to turn. For teams working on long-context, RAG, or multimodal retrieval, this is a replicable "locate first, train second" recipe; the next time an omni-modal model finds evidence quickly and accurately, it may well be a few dozen heads doing the work.