AI peer review is no longer a demo; it runs at production scale. But who audits the auditors? Five researchers from the University of Cambridge, the Flatiron Institute and collaborating institutions turned 30 real arXiv papers into an exam: they planted 357 graded errors into the papers and let 12 AI reviewer systems hunt for them. All numbers below are the paper's own measurements. The work has been accepted at NeurIPS 2026 (https://arxiv.org/abs/2611.17052).

How the exam is built

Two corpora of 15 papers each: an astrophysics set drawn from astro-ph submissions of 2026-04-22, and a machine-learning set sampled from cs.LG submissions of 2026-07-15. Both days postdate the knowledge cut-off of every evaluated model, so nobody can answer from memory. Errors come in four types — general issues, mathematics, numerics, citations — at three severity levels, injected by agents directly into the LaTeX source and accepted only after a mechanical validator passes. Verdicts come from a cross-family panel of Anthropic, OpenAI and Google models by majority vote, while clean copies are used to measure false positives.

Choosing the model matters more than the architecture

Given the same one-sentence prompt, GPT-5.5, Claude Opus 4.7 and Gemini 3.1 Pro reach total detection rates of 60.5%, 36.7% and 11.9% on astrophysics — a 48.6-point spread; on ML the numbers are 63.9%, 43.9% and 11.1%. Compressing the multi-agent system Skepthical's full operating instructions into a 978-line structured prompt lifts the weaker models: Gemini gains 25.4 to 40.0 points, Claude gains 16.6 to 28.3. Strong configurations still catch 57% to 73% of even the subtle injections, while weak ones miss most errors even when blatant.

Tools get used only when they are enforced

Claude Code and Skepthical both hold arXiv access, yet the execution traces show Claude Code never actually retrieved a cited paper. Skepthical makes retrieval a mandatory step and reaches a 46.7% citation-error detection rate on astrophysics, against at most 26.7% for any other system. With model and sandbox fixed, swapping the one-sentence prompt for the structured instructions alone raises Claude Code's total detection by 32.2 points on astrophysics and 38.9 on ML — the structured configuration wrote 21 Python verification scripts, the free-form one wrote none.

Two surprises

First, the two open-weights reviewers fine-tuned on review data detect almost nothing on their home ML corpus — 1.7% and 2.8%. Second, mathematics flags are especially unreliable: pooled over frontier systems, 30% of math flags on astrophysics and 67% on ML are judged false, and math flags account for about half of all false positives. Otherwise false positives stay contained: apart from the two open-weights reviewers, every system is below 9% on astrophysics, with Skepthical at 2.8% and 4.0% across the two corpora.

Beyond the scores

The ceiling is modest: Skepthical detects 70.6% on astrophysics, still missing roughly a third; on ML, structured single-model GPT-5.5 actually leads at 81.1%, a difference the paper calls not significant. The authors also self-discount: Skepthical draws on three providers and is not a capacity-matched comparison, so its edge cannot be credited to the pipeline design alone. On cost, hosted systems span 0.34 to 2.41 dollars per paper: Skepthical at 2.41 and about 24 minutes per paper, the structured single-LLM configuration at 0.55 and about 2 minutes. The evaluation pipeline, the injection catalogues and the full adjudication logs are open-sourced (https://github.com/inigozubeldia/skepthical-benchmark).

So what

For venues rushing to deploy AI reviewers, this benchmark offers a ready-made template: run an error-injection audit before launch, treat model choice as the main lever, and report false positives separately — math flags above all. The lesson generalises to every agent deployment: make verification optional and the agent will skip it; make it unavoidable and it finally starts reading the literature.