Public benchmarks have been broken for a while now. Test sets circulate online, vendors train on them, leaderboard scores rise, and actual capability may not move at all. Everybody knows this. But nobody wants to change the rules first, because changing the rules means admitting that the leaderboards you published earlier did not really mean much.

Artificial Analysis just did it. The Intelligence Index v5, published on October 27, is what the company calls its biggest overhaul since v4 launched in January, and the core move is a single one: 60% of the weighting now sits on evaluations with private test sets or held-out answers. The stated reason is blunt: benchmark contamination. Once questions and answers circulate, scores get inflated by training data. The full announcement is at the official Artificial Analysis post.

Two New Yardsticks, Three Repairs, Three Deletions

Two benchmarks are new:

  • FrontierCode 2.0: Cognition private coding benchmark, 132 tasks drawn from real open-source repositories. The grading question is not "is the code correct" but "would a maintainer actually merge this PR". Failing any single blocking check scores zero.
  • Terminal-Bench-Science: 70 agentic scientific research tasks run in a terminal environment. A task only passes if every one of its tests passes.

Three are upgraded: HLE-Diamond (a refined 1,000-question version of Humanity's Last Exam from CAIS and Scale AI, split evenly between reasoning and knowledge), AA-Omniscience v1.1 (saturated and public-set questions removed, new questions added, ambiguous ones revised), and a hard subset of CritPt v2 (70 research-level physics questions minus the 30 that seven frontier models solved in all five attempts at launch, leaving a private 40).

Three are removed entirely: GDPval-AA v2.1, SciCode, and AA-LCR v1.1.

The weighting is now four equal blocks at 25% each: Knowledge Work (AA-Briefcase 15%, AutomationBench-AA 5%, GDP.pdf 5%), Coding (FrontierCode 2.0 15%, Terminal-Bench 4.0 10%), Science (Terminal-Bench-Science 15%, CritPt v2 hard subset 10%), and General (AA-Omniscience v1.1 15%, HLE-Diamond 10%).

A Counterintuitive Curve

Cognition published a data set that is rare to see. On FrontierCode, turning reasoning effort all the way up can lower the score.

On FrontierCode 1.1, Claude Opus 5 implementation criteria rise from 79.3% on low effort to 83.0% at max effort, while scope criteria fall from 69.2% to 51.1%; GPT-5.6 Sol behaves the same way, with scope dropping from 79.7% to 68.3%. Put plainly: when you let the model think more, it edits more files and touches more code, and those edits frequently go beyond what should have changed. Cognition attributes this to models overdoing it: they reach for unrelated implementations on the side, and scope collapses as a result. Two contributing factors come along for the ride, namely that higher effort makes it more likely to accidentally pull in the solution commit, and outright overthinking where a model talks itself into implementing a misreading of the requested change.

FrontierCode 2.0 loosened the scope criteria, and the average scope failure rate drops to 3.8%, down from 9.0% in 1.1. Overall strictness went up rather than down. Thoroughness blocking criteria grew from 168 to 501, tasks are drawn from 55% more repositories, blocking criteria count 43% more, and reference solutions modify 67% more files and 89% more lines of code on average. In 80% of FrontierCode 2.0 tasks the reference solution differs from the upstream PR, up from 60% in 1.0. And in 2.0, thoroughness becomes the category with the highest failure rate at every effort level, at 26.0% on low effort and 15.3% at max effort.

Harder Scores, But No Longer Independently Checkable

Privatizing the tests fixes contamination and costs you outside verification of the numbers. AlphaSignal put the tradeoff plainly in its coverage: restricted access does reduce the chance that benchmark material leaks into training data, but it also limits independent replication.

The same benchmark, two harnesses, and the numbers can diverge by several points. The Cognition leaderboard gives DeepSeek V4.1 Flash 19.3%, while Artificial Analysis rerunning it in its own Stirrup harness reports 28%, a gap of nearly 9 points.

The leaderboard also got more expensive. Among the ten highest-scoring models, cost per Intelligence Index task ranges from $0.96 for GPT-6.1 Terra to $13.04 for Claude Fable 5.1, a spread of more than 13x. Median output tokens per Index task rose by two thirds, from 35k to 59k. Agentic evaluations account for 55% of the Index weight but roughly 90% of output tokens, and Terminal-Bench-Science alone accounts for about a third of output tokens per task.

The open weights gap also got measured by the new ruler. The distance between the leading closed model and the top open weights model grew from 3 points in v4.3 to 12 points in v5. Counting only models whose weights are actually downloadable, the gap reaches 26 points. DeepSeek V4.1 Flash 1017 and GLM-5.3 both score 32, and closed-source Claude Haiku 5.5 at 36 outscores every one of them.

After Changing the Ruler, Bring Another One

v5 scores are not directly comparable to v3 or v4, since changing the questions shifts the scale of the scores. So on the same day Artificial Analysis also shipped Progress Index v1, which uses item response theory from psychometrics to place every model released since November 2022 on one continuous capability scale. The anchors are o3-mini at high effort set to 220 and GPT-5.5 at extra-high effort set to 320, with no percentage unit. The methodology cites prior work from the Epoch Capabilities Index and Anthropic capability-scale research.

A leaderboard is not a fact. It is a measurement act. Changing the ruler does not change what the models can do, only what you are able to see of it. So read a score together with how it was measured, read a cost together with how far the token bill moved, and when you read an open weights gap, ask the sharper question: did the models change, or did the benchmark get harsher?