In a beekeeping clip where you can only hear crowing, an omni-modal model will tell you there is a rooster in the picture. Flip it around: a shot of fabric being sprayed with someone speaking over it, and the model may flatly deny that anyone was talking. Nothing was misperceived. Another modality's prior simply won the argument inside the fused representation: you asked about vision, it answered from ears.

The fault is not in the output layer, it is in the fusion layer

Most of the last two years' work on multimodal hallucination stays at the output level. Contrastive decoding penalises tokens that remain likely when the perceptual input is degraded. OPERA adds an over-trust penalty and reallocates attention after the fact. The controlled baselines here, MAD and HulluEdit, live at the same level. A paper from Nanjing University and the University of Chinese Academy of Sciences argues that this misses the point: early fusion entangles grounding evidence with learned priors inside one fused representation, so nudging the output just stirs an already mixed signal again.

The paper isolates a diagnosable failure class called cross-modal hallucination: evidence from one modality is overridden by a cue from another modality, or by the language prior. The CMM benchmark describes these failures as language, visual, and audio dominance plus spurious cross-modal correlations. AVHBench tests audio-driven visual hallucination, visual-driven audio hallucination, and audio-visual matching.

Three steps: find the conflict, split the evidence, suppress the prior

The method is called Omni-Woodpecker, OWP for short, and it does not change a single weight.

Step one is finding the conflict. The same frozen model is run on four views: the full input (query plus video plus audio), audio only, video only, and text only. The text-only view acts as a shared reference that separates modality evidence from language-driven answers. If the full view and the target-modality view agree, there is no conflict and OWP returns the backbone answer untouched. When they disagree, a cross-modal conflict exists, and the method works out which modality the query asked about and which one answered instead.

Step two is splitting the evidence. Orthogonal decomposition on the hidden states separates the grounding evidence from the competing prior into two directions. Step three is suppressing the prior and restoring the evidence: before decoding, one correction budget is split between token-head path reweighting and residual-state restoration, and the edited state is written back to the same layer. The ablations show the residual state is the main site of correction, with evidence best restored in middle layers and priors best suppressed in later layers.

Target typing is worth a mention. OWP appends a diagnostic query asking whether the question needs audio, video, or both, takes the softmax over three next-token logits to build the scores, and the paper reports it agrees with the benchmark-defined target modality on 99.48% of rows.

The numbers: +12.9 on CMM, and it transfers across architectures

With Qwen2.5-Omni-7B as the backbone, CMM accuracy goes from 73.2 to 86.1 and AVHBench from 75.1 to 83.7. The abstract reports +12.9 and +8.6 points, and the paper reports gains over the MAD baseline of 4.9 and 7.7 points on the two benchmarks. The more interesting number is hallucination resistance: HR climbs from 57.0 to 85.5 while perception accuracy lands at 86.7. The backbone had a 32.5-point gap between PA and HR; OWP compresses it to 1.2 points, where MAD still leaves 16.9.

On VideoLLaMA2-AV, a backbone with a completely different audio-visual fusion design, CMM goes 76.2 to 80.8, AVHBench 70.2 to 74.1, and HR 65.3 to 77.7. The paper also reports that OWP on Qwen2.5-Omni-7B beats the newer Qwen3-Omni by 4.3 and 6.2 points.

General omni-modal reasoning is not sacrificed. OmniBench moves 45.36 to 47.29, WorldSense 45.21 to 47.46, MUSIC-AVQA 85.08 to 86.08, and AVQA 91.83 to 92.50. The statistical setup uses the 4,604 examples on which every method returns a valid prediction, with exact two-sided McNemar tests and 95% confidence intervals from 5,000 row-level bootstrap resamples; the paper reports the lead on the common subset at p<10^-16.

The bill is printed openly: 1.71x latency, 24.9 GiB of memory, open weights only

The paper measures its own inference cost on an RTX A6000, averaged over three runs. OWP costs 6.05 seconds per input on Qwen2.5-Omni-7B, 1.71x the backbone's 3.54 seconds, with at most five forward calls. MAD is 9.96 seconds, 2.81x, and nine calls. HulluEdit is 13.02 seconds, 3.68x. Peak memory on the two backbones is 24.9 GiB and 17.3 GiB. OWP is cheaper than both baselines because of the conflict-triggered schedule: only 25.8% of inputs are judged to contain a conflict and 16.9% have their final answer changed, while the remaining 74.2% run the unmodified backbone.

Two costs are harder. First, OWP edits hidden states and attention outputs, so it needs access to model internals, and the paper states plainly that it applies to open-weight OLLMs but not to models available only through an API. Second, it restores evidence the backbone already encoded but failed to use during fusion; if the target modality is misperceived even on its own, there is no reliable evidence to restore. The authors also concede that it covers only the cross-modal class, leaving temporal hallucination, speaker attribution, emotion, and prosody for later.

So what

The value here is not another hallucination-suppression method but a relocated problem: hallucination is not about what the model says, it is about which signal inside the model drowned out which other signal. And the training-free constraint draws the ceiling in return, because you have to reach inside the model, and you have to accept the extra 1.7x latency and 24.9 GiB of memory.

The code ships under the MIT license at github.com/ZiiJianLiu/Omni-Woodpecker, the paper is arXiv 2610.27103, submitted on October 19 and ranked third on Hugging Face's Daily Papers board on October 26. The README asks for at least 24 GB of GPU memory in BF16, with weights cached under ~/.cache/omni-woodpecker by default.

If the next fight for omni models is about saying less wrong rather than saying more, moving the intervention point from the output layer down into the fused representation may beat another round of post-training.