Large vision-language models get a dedicated score for saying things that are not in the image. Preference-optimized checkpoints have pushed that hallucination score down for years, and the field has quietly treated the problem as solved. The trouble is that the same intervention also changes how much the model is willing to say, and those two effects have been scored together. This paper (arXiv:2611.23179, https://arxiv.org/abs/2611.23179) takes four public preference-optimization checkpoints apart and asks how much of the hallucination drop comes from the model actually seeing better, and how much comes from it shutting up earlier.

One table, two contradictions

The study uses LLaVA-1.5-7B and four public preference-optimized checkpoints of it: RLAIF-V, CHAIR-DPO, CSR and HA-DPO. On CHAIR-500 all four models lower CHAIRi, and all four also lower object recall. The base model recalls 80.5 percent of ground-truth objects; the four checkpoints land between 65.1 and 76.2. Caption length does not shrink the same way: the base caption runs 89.7 words and the checkpoints keep 75 to 93. But the number of distinct object categories named per image drops from 3.2 at the base model to between 2.1 and 2.8. Hallucination rate and coverage fall together.

The complication is that two ways of counting hallucination point in opposite directions. The paper also reports a deduplicated hallucination rate, Hu. On CHAIR every checkpoint sits 4.9 to 19.1 points below the base model on Hu, and 1.6 to 6.9 points below on AMBER, with all eight intervals excluding zero. But set-level object F1 on CHAIR shows no significant gain for any of the four, and it drops for CSR on the held-out set. The difference comes from repetition: on CHAIR, 57 to 65 percent of object mentions repeat a category the same caption already named, and on AMBER it is 49 to 54 percent. CHAIRi counts those repeats, Hu does not.

Making the base model shut up

The key experiment is simple. Take the base model, add a positive bias to the EOS logit, and let it stop earlier until its recall matches the checkpoint. On two disjoint sets of 500 COCO images, this early-stopping curve reproduces 58 to 86 percent of the CHAIRi gain of RLAIF-V, CHAIR-DPO and CSR, which works out to 61, 77 and 86 percent per model. No training, no weight changes: make the original model say a few fewer sentences and most of the reported hallucination improvement comes back.

It is not a length effect either. Matching on word count does much worse, with the same bias reproducing only 0 to 24 percent of the gain. And at equal recall the checkpoint captions run 1.3 to 1.8 times longer than the ones on the stopping curve, so a length-controlled comparison misses the effect entirely.

Writing longer does not bring it back

The opposite move is to make the checkpoints talk more, by adding a negative bias to their EOS logit and forcing them to continue up to a 512-token cap. They recover only 2 to 9 percent of the lost recall. On CHAIR, RLAIF-V, CHAIR-DPO and CSR recover 2, 9 and 4 percent; on AMBER, 17, 24 and 8 percent. Between 94 and 96 percent of the newly added object mentions repeat categories the caption already named. What got dropped does not come back just because the model keeps writing.

The base model, though, still remembers those objects. On the CHAIR development set, the base caption names 47, 48 and 47 percent of the ground-truth categories that RLAIF-V, CHAIR-DPO and CSR respectively miss, while the forced-longer checkpoints name only 2, 13 and 2 percent. On AMBER the base caption covers 18, 15 and 16 percent of the missed objects against 2, 6 and 1 percent at the matched setting. So the paper proposes a training-free fix: fuse the checkpoint caption with the base caption, appending only the categories that the base model mentions, the checkpoint omits, and the base model is confident about.

The threshold is fixed on the development set and applied unchanged to the held-out set. The restored captions of RLAIF-V, CHAIR-DPO and CSR reach recall of 79.2, 79.5 and 79.7 against 80.1 for the base model. Their Hu values are 21.6, 25.0 and 25.8 against 26.9, so restoration pushes Hu below the base model by 5.24, 1.85 and 1.08 points, with intervals excluding zero. The cost is that it keeps only part of the checkpoint gain: of the categories it appends on the held-out set, only 41 to 45 percent are correct.

Three independent lines of evidence

Saying less is not the same as seeing better, and that reading is not unique to this paper. Fazli and colleagues (https://arxiv.org/abs/2609.01888) evaluated six mitigation methods across three LVLMs and four benchmarks, and concluded that hallucination reduction is often coupled with reduced informativeness, with current evaluation protocols potentially overestimating progress by rewarding conservative generation. Sandhu and colleagues (https://arxiv.org/abs/2608.24966) targeted the attention heads behind object hallucination in LLaVA, lowering CHAIRi from 0.156 to 0.096 while object recall fell from 0.78 to 0.70. On the mechanism side, Yue and colleagues at ACL 2024 (https://aclanthology.org/2024.acl-long.633/) trained models to terminate generation on time, cutting CHAIRs and CHAIRi by 26.4 and 26.6 percent at the cost of 1.8 points of recall.

The cold water

The paper is candid about its edges. It studies one backbone, LLaVA-1.5-7B, with greedy decoding and a single detailed-caption prompt, and its two COCO sets are disjoint but come from the same dataset. Attribute and relation hallucination, along with checkpoints the authors could not validate, are out of scope. All intervals are nominal 95 percent with no correction for multiple comparisons. The authors state plainly that the stopping share is a reproduction ratio and that they make no causal claim about the training effect. The paper says it will release the evaluation code, but gives no repository address.

So what

Scoring hallucination on one number alone is no longer trustworthy. The practical rule is to always report object recall alongside the hallucination rate, and to compare at equal recall, because otherwise a model that learned to stop early looks like an improvement. The paper also turns up a second point: what a model stops saying is often not something it forgot. Its base knows, and preference optimization just pushed it down. So the next time a model release shows a large drop in hallucination rate, what is the first number you would ask for?