When a multimodal model counts three cups in an image as two, you can't easily tell whether it failed to see or failed to reason. Almost no existing visual benchmark answers that question — they test end-to-end answers and mix perceptual errors with reasoning errors on the same scoreboard. Moonshot AI has now split the two apart with PerceptionBench, a benchmark built specifically for "atomic visual perception." The dataset is on Hugging Face (CC-BY-NC-4.0), the evaluation code is open-sourced under Apache 2.0 on GitHub, and the methodology is described in arXiv:2607.24957.
Built From 42 Benchmarks' Failure Points
The most interesting thing about PerceptionBench isn't the scores — it's how the questions were made. Instead of pre-defining what "visual perception" should test, the team traced frontier models' earliest failure points across 42 existing benchmarks, attributed those failures to perception, and distilled the attributions into ten atomic capabilities: visual relation, counting, attribute, depth & 3D, localization, comparison, fine-grained recognition, context integration, OCR, and perception-related hallucination. Each question tests exactly one capability, with short, uniquely determined answers — difficulty comes from seeing, not from reasoning or knowledge. The released set of 3,000 verified questions was subsampled with capability-level balancing and difficulty stratification from an in-house pool of 17,000+ verified samples; 60% are atomic sub-questions decomposed from attributed failures on source benchmarks, and the remaining 40% were newly authored on supplemented images.
Sixteen Frontier Models, None Clears 60%
The results are rough. Sixteen frontier multimodal models (ten proprietary, six open-source) ran the full set with unified prompts and the highest available reasoning budget — and none reached 60% overall accuracy. GPT-5.6-Sol leads at 59.7, Kimi K3 takes second at 58.5, Claude-Fable-5 third at 57.2, Gemini-3.1-Pro 56.2, GPT-5.5 55.8. The best open-weight performer is Qwen3.7-Plus at 51.1; in the lower half, Grok-4.5 manages only 41.0, GLM-5V-Turbo 39.6, Minimax-M3 33.1, with GLM-4.6V at the bottom on 32.5. Scoring used GPT-oss-120B as judge, with 99.7% agreement with human judgment on a 300-sample audit.
The subscores carry the real information. Models with nearly identical overall scores can have completely different capability profiles: GPT-5.6-Sol scores 76.7 on localization while Gemini-3.1-Pro gets 52.7 on the same category, yet their overall scores differ by just 3.5 points. The overall leader GPT-5.6-Sol scores only 26.9 on perception-related hallucination — it frequently "sees" things that aren't there — while Gemini-3.5-Flash, sitting at 52.0 overall, scores 50.6 on hallucination. None of this is visible on a single leaderboard.
Guessing Right Isn't Seeing
The paper also contains a quiet but important observation: a large share of correctly answered questions don't survive being asked again. Models are often pattern-matching rather than perceiving — inferring what an image probably contains from language priors instead of reading the pixels. For anyone selecting models on holistic benchmarks like VQA or MMBench, this is a warning: models with strong language ability can compensate for weak perception with priors and "test" their way to a higher overall score. There's also a structural problem: the perception-error slices captured by existing benchmarks overlap only weakly, with a mean pairwise weighted Jaccard of 0.20 — no single benchmark, or small group of them, approximates perception as a whole. You can't assemble a perception axis by stapling together a few existing evals.
So What
For teams building multimodal systems, PerceptionBench works best as a diagnostic: a model scoring 55 overall but ~30 on depth perception tells you exactly where to invest in training data or architecture. For everyone else, the leaderboard clarifies one thing — the bottleneck of frontier models may not be reasoning, but the "seeing" upstream of it. If even the top-scoring model sees things that don't exist, would you trust it with radiology reads, factory inspection, or surveillance monitoring?