Counting has long been treated as a soft spot for vision-language models — they miscount, and nobody can say by how much. Someone finally measured it the way you calibrate a sensor: a new paper puts eight open VLMs to work counting vehicles in aerial imagery and finds a hard ceiling.

The ceiling is about five objects, not a vague "it miscounts"

The authors cut 960 square crops from 260 DOTA-v1.5 training images, 60 crops for each of eight count bins, running from one or two objects up to more than 160. None of the crops come from the benchmarks they later evaluate on. Each crop holds a single class — cars, trucks, ships or storage tanks — and the model is asked how many it contains.

For Qwen3-VL-8B the error stays within 15% up to five objects, turns systematically negative beyond that, and reaches -57% above 160 objects. Past the ceiling the answers also start settling on round numbers, the way a sensor does once it goes past its range.

They call this the counting horizon. The eight models come from three families, and each has its own onset: Qwen3-VL-2B begins to fail from the three-to-five bin, Qwen3-VL-4B and 32B together with Qwen2.5-VL-7B and LLaVA-OV-7B all start at six to ten, while the two InternVL3.5 models hold on to forty-one to eighty — but they overcount small sets rather than counting accurately. The sharpest case is Qwen3-VL-8B: on its first set of crops the onset is six to ten, but on an independently drawn second set it becomes three to five, so the same model's turning point moves by a whole bin.

The ceiling is not a constant either; it moves with the view. For the same 160 objects, cut into 256-pixel crops and shown enlarged threefold, the error stays within 21%. Shown at 2048 pixels at native resolution, every count bin falls short. Other imagery behaves differently again: FAIR1M cars slip past 20% only beyond 20 objects, DIOR ships only above 160, and on VisDrone drone imagery the error stays within 15% up to 40 objects.

The ceiling can be computed

Rather than stopping at a description, the paper fits a power-law response model: c(n) = a·n^γ, with γ = 0.76 for Qwen3-VL-8B on its first crop set and between 0.43 and 0.80 for the others. Since γ is below one, the answer is always only a discount on the true count.

From that, the paper derives what splitting a view is worth. Split a view into K parts and the summed answer is amplified by A = Σ(n_k/n)^γ over the original, with 1 ≤ A ≤ K^(1−γ). The validation holds up: for Qwen3-VL-8B the predicted and observed amplification agree within 0.08 in every bin from 6 to 160 objects; on a second, independently drawn set of crops, within 0.10 in four of five bins; for the seven other models the observed value lies mostly within 20% of the prediction.

The cost is stated too. An empty view is answered with 0.8 to 2.4 spurious objects on average, so the finer the split, the more phantom counts accumulate. Above 160 objects the observed amplification is about 2.1, far above the predicted 1.3 — at that scale a second limit kicks in, because some objects are simply too small to see in the original view.

No training, the same frozen model

GeoScope works like this: a text-only model parses the question into a typed spatial program that fixes the region and its resolution; the VLM only grounds and counts within its operating range; directions are computed from the grounded boxes instead of being asked of the model.

The results: regional counting on XLRS-Bench goes from 47.50 to 63.75, spatial relations from 33.50 to 44.00, and MME-RealWorld counting from 25.18 to 36.60. On UHR-Micro, global counting reaches 40.49 and regional counting 49.34, which are +16.8 and +36.2 against the authors' own re-run baseline, with 15.6 of those points coming from splitting only when a view exceeds the horizon. Conditional counting is the exception at 17.48, six points below that baseline. Both direction and distance reach 100.00, since those questions already mark the two objects and the answer is computed from geometry rather than from the model.

The bill shows up in time and tokens. Spatial relations take a median 17.2 seconds and a median of 11.0k visual tokens, against 2.8 seconds and 3.8k for zero-shot.

The place where the paper's own ledger does not balance

More interesting than the scores is the failure analysis.

On XLRS-Bench overall counting, GeoScope lifts zero-shot answering from 10.42 to 35.42, which looks like a rout. But the paper itself states that this score does not beat always picking the second-largest of the four numbers, which scores 37.50.

The reason is the answer options. True counts in that task have a median of 61.5 and reach 734, adjacent options sit about 17% apart, and picking correctly requires a count within 8% of the truth. GeoScope's counts there carry no systematic bias but scatter by 18%, and only 9 of the 48 fall within 8% of the truth.

A second trap in the benchmark is subtler. The option stating that the image does not show the number is correct for only 5 of the 1226 counting questions, yet backbones pick it for 18 to 57% of the test questions, and the reproduced methods for 9 to 18%. The paper therefore re-runs each backbone restricted to the numeric options — and that restriction alone significantly beats every reproduced method on the whole test split.

On spatial relations, 86% of wrong answers point more than 45 degrees away from the correct option, which says the dominant error is grounding the wrong object rather than counting wrong. A manual check of 150 programs shows that hand-written programs change accuracy by at most two questions out of thirty per task.

So what

The transferable result here is turning "models miscount" from an adjective into a calibratable engineering parameter: where the ceiling sits, what moves it, how much a split recovers, and what the split costs.

The second transferable result is blunter. A benchmark that can be beaten by always picking the second-largest number is not measuring counting ability — it is measuring option spacing. The paper says as much in its own discussion: to tell counting procedures apart, a benchmark needs questions spread across the whole range of counts, options spaced in proportion to the count and placed at random around it, and any option that is almost never correct dropped.

Claims like "the first UHR method to split views by a measured limit of the VLM" remain the authors' own assertion so far, without independent third-party corroboration.

Original paper: arXiv:2611.12699; project page: GeoScope.