Multimodal leaderboards just got a crack in them, and the crack comes from an arXiv preprint.
The paper is called RAMP. It went up on 16 November 2026 (https://arxiv.org/abs/2611.20653), authored by researchers from the University of Science and Technology of China, Central South University and Yanshan University. What it does is deliberately unglamorous: it isolates one step of multimodal evaluation that almost nobody scrutinizes — the answer readout protocol, meaning how the evaluation program actually pulls an answer out of the model's output.
The headline finding is blunt. Same image, same question, same model weights, and simply changing the interface used to read the answer shifts leaderboard scores by up to 20 percentage points.
Where the gap comes from
On 500 adversarial POPE questions, the paper runs three backbones — SmolVLM-256M, TinyLLaVA-1.1B and Qwen2.5-VL-3B — and compares four readout families: bare (score the raw yes/no tokens directly), surface-max (fold in capitalization and other single-token spellings and take the highest), typed (show the model 'A. yes' and 'B. no' and score the option letters), plus a reversed variant that flips the option order.
SmolVLM-256M falls from 71.8% under bare scoring to 51.6% under typed, a 20.2 point drop with a 95% paired interval of [-27.6, -13.2]. TinyLLaVA-1.1B goes from 82.4% to 66.0%, a 16.4 point gap. Qwen2.5-VL-3B moves from 85.2% to 87.0%, only 1.8 points. The larger SmolVLM-500M, on its own CPU configuration, drops from 82.2% to 60.0% — 22.2 points.
The pattern is plain: small models are extremely fragile under readout protocols, while the ones with more parameters hold steady. That contrast is more interesting than the familiar 'prompt wording changes the score' story, because it suggests many small-model scores were never trustworthy in the first place. Swapping to the surface readout also does not push everything one way — SmolVLM-256M actually climbs from 71.8% to 75.0% while TinyLLaVA drops from 82.4% to 78.4%.
Swap the protocol, change every answer
More unsettling is what the authors found about the aggregate number itself.
In the archived SmolVLM-256M POPE run (counted separately from the newly executed batch above), switching from bare to typed moves 221 items from correct to incorrect and 121 from incorrect to correct — 342 of 500 items flip correctness entirely, while the net effect is only -20.0 points. TinyLLaVA records 119 correct-to-wrong and 38 wrong-to-correct transitions, 157 changed items, net -16.2 points. Qwen changes just 33 items.
The starkest case sits on a 104-question native yes/no subset of ChartQA: SmolVLM-500M has 40 wrong-to-correct and 39 correct-to-wrong transitions. Seventy-nine answers change, 76.0% of the subset, and the final accuracy moves by only +1.0 point.
In other words, a leaderboard score that looks stable can be two entirely different answer sheets cancelling each other out. Reverse the A/B option order and 401 of SmolVLM-256M's semantic predictions flip; for Qwen, only 7 of 487 jointly observed reversed items change.
Swap the protocol, and confidence collapses too
The paper also measures calibration. SmolVLM-256M's 15-bin correctness ECE rises from 0.221 to 0.409, and SmolVLM-500M's from 0.093 to 0.238; TinyLLaVA goes from 0.061 to 0.084. Qwen moves the other way, from 0.065 down to 0.050. AUROC tracks along: SmolVLM-256M drops from 0.7011 to 0.6084, and Qwen from 0.7927 to 0.7762.
Prompt wording is just as sensitive. One formulation lifts SmolVLM-256M by 13.2 points relative to the original prompt, while capitalizing the answer token costs 10.2 points; TinyLLaVA loses 17.2 points on the capitalized variant. Qwen's column stays inside a -0.4 to +2.0 band. A separate 64-question experiment on whether the options sit before or after the question produces a -23.44 point result for TinyLLaVA.
Same code, different machine, different number
The authors did something few of their peers do: they archived raw logits, candidate token IDs, semantic mappings, precision and image policy, then independently replayed the evaluation from that stored evidence. They reconstruct 16,140 first-token decisions from the JSONL and match every processed prediction exactly; a second pass over the 57,073-row readout table reproduces all 217 exported accuracy cells.
The replay also surfaced an uncomfortable fact. Two SmolVLM-256M POPE runs on 500 questions produced typed-minus-bare changes of -14.4 points on CPU/fp32 and -20.2 points on MPS/fp16. The semantic predictions were identical on all 64 control questions, yet individual score margins differed by as much as 0.136694. The magnitude of the conclusion depends on what hardware you ran it on.
Not a new mechanism, but a new level of measurement
That readout style changes conclusions is not itself new. The 2024 paper Order Matters (https://arxiv.org/abs/2410.16983) showed that multimodal large language models are acutely sensitive to input order, that reordering can swing performance between near-best results and random guessing, and proposed Position-Invariant Accuracy to correct for it. A June 2026 audit, Facet-Probe (https://arxiv.org/abs/2606.26079), perturbed 18 frontier and open-weight multimodal models across five facets — option, evidence-chunk, document-rank, image-set and mixed-modality ordering — and found none of them order-invariant, with per-facet flip rates spanning 24% to 50% and the best model in that audit still flipping on 13.4% of trials. Earlier still, State of What Art? (https://arxiv.org/abs/2401.00595) demonstrated the brittleness of single-prompt evaluation across 6.5M instances, 20 LLMs and 39 tasks, and was accepted at TACL.
What RAMP adds is decomposing the effect into a recomputable protocol layer, with a complete audit trail behind it.
So what
A chunk of any multimodal leaderboard number does not belong to the model. Within a single evaluation, an engineering detail like 'ask yes or ask A' can eat twenty points off a small model's score.
If you are shipping a model, stop reporting bare accuracy — report the readout protocol and the calibration alongside it. If you are reading a leaderboard, ask one question first: which interface was this score extracted through?
A model has to be read correctly before it can be evaluated. Read it wrong, and the score is just a misunderstanding.