SLAM-ASR is a famously lazy recipe for speech recognition: freeze the speech encoder (Whisper turbo), freeze the large language model, and train only a small projector — a two-layer MLP with hidden size 2048 that concatenates every 5 consecutive encoder states — to wire the two together. The original paper literally calls the approach embarrassingly simple, and it still yields competitive transcription. One question, however, stayed unanswered: why is a small trained map enough? Artem Fedorchenko and Tanel Alumäe of Tallinn University of Technology, Estonia, now give a circuit-level answer (https://arxiv.org/abs/2611.24721): the components doing the real work are the induction heads inside the frozen LLM.

Induction heads: the copying circuit LLMs already have

An induction head is one of the best-studied circuits in text LLMs: if it has seen A followed by B, then on the next occurrence of A it attends to B and raises the logit of B — prefix matching plus copying, in one head. Anthropic's Olsson et al. described the mechanism systematically in the 2022 Transformer Circuits series (https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html), and an independent team later confirmed induction heads as an essential mechanism for pattern matching in in-context learning at NAACL 2025. What is clean about the new paper is how it finds the heads: it never looks at speech. It feeds each model 40 sequences of 50 random vocabulary tokens repeated twice, computes a prefix-matching score per head, and keeps the top 3 percent as the set Ck — 20 heads in Llama 3.2-3B, 6 in Gemma 2-2B, and 25 in Ministral 3-3B.

Block ten heads and the error rate doubles

The causal test is attention blocking: the selected heads' attention logits on speech positions are set to negative infinity, forcing each head's attention to redistribute over the rest of the context. On Llama 3.2-3B, blocking the top 10 induction heads pushes word error rate (WER) from 3.3-3.6 up to 7.1-8.3 — roughly doubled — while blocking 10 random heads drawn from the same layers leaves WER essentially unchanged. Blocking the top-10 set raises WER in all three decoders. The paper's reading: speech embeddings play the role of the first occurrence in the sequence. Each word's speech embeddings approximate its generated token, so from the just-generated word, an induction head attends to the speech embeddings of the word that comes next and copies it out — the same old circuit, rerouted through audio.

Pairs of heads: differential attention that grew on its own

The more striking find is a pairing structure. In each layer the authors take h1, the head with the largest ASR importance, and h2, the head of the same layer whose output direction is most anti-aligned with h1. Some of these pairs share a single key/value group under grouped-query attention: both heads attend to the same positions, but their outputs point in opposite directions, so the shared attention noise cancels and attention sharpens onto the previously generated word. That is precisely the idea of the Differential Transformer from ICLR 2025 — except here nobody designed it; the model grew it. Blocking h1 alone hurts far more than blocking both heads together, while blocking h2 alone leaves WER unchanged in Gemma and Ministral.

A noise-reduction switch, picked up for free

What happens if you amplify these heads instead? At inference time the paper scales the outputs of Ck by 1.5, and evaluates on 400 test-clean plus 400 test-other utterances with babble and white noise added at 5 and 0 dB SNR. On Llama, in the hardest condition — babble at 0 dB — WER drops from 89.0 to 81.8 and the insertion rate from 40.4 to 32.1; the mean WER under noise goes from 41.9 to 39.9. In the control, scaling the remaining heads of the same layers pushes noisy-mean WER up to 50.8, worse than not scaling at all, and across 3 random seeds Ck scaling is consistently better than scaling random heads. The cost on clean audio is at most 0.2 points. An independent evaluation at ICASSP Workshops 2025 had already found that SLAM-ASR degrades significantly under additive noise, which makes this a cheap patch. Note, though, that the trick works in only 2 of the 3 LLMs — it does not help on Gemma.

The cold water

Every number here is self-reported by the paper; training used only the clean-100 subset of LibriSpeech, and the noise evaluation covers 800 utterances. How the projector should be trained, and whether the heads should be updated alongside it, remains an open question in the paper — on its own account, today's projector merely smears speech into a noisy and redundant version of the transcript, so the LLM's old circuits can just about read it.

So what? The value of this paper is not another benchmark point. It is an honest audit of the freeze-and-reuse route: you are not training a speech recognizer, you are wiring up circuits the LLM already has. The mechanistic explanation converts directly into an inference-time intervention — one scaling factor buys a few points of noise robustness for free. The next time you see a paper that aligns a modality by training one percent of the parameters, ask the obvious question: on the frozen side, which heads are actually doing the work?