Model evaluation looks deceptively simple: run a model on a set of questions, average the result. The trouble sits precisely in that word "average" — a benchmark score is a weighted average over some distribution of difficulty, and nobody has been checking what that distribution looks like.
A paper accepted at NeurIPS 2026 tears the wrapper off: Your Benchmark Is an Empirical Measure Over Difficulty (arXiv:2610.29927). Its authors, from the University of Illinois Urbana-Champaign, UC San Diego, and New York University, formalize a benchmark as an empirical measure over an axis of difficulty: each of n items contributes equal mass 1/n at its own difficulty z_i ∈ [0,1], so the score of a model becomes ∫ s_θ(z) dν_B(z). Once written this way, two failure modes that were rarely named before come into view.
Two failures: distributional mismatch and coverage holes
The first is measure mismatch: the difficulty distribution of the benchmark differs from the distribution it was meant to test, so leaderboard conclusions can diverge from conclusions under the target distribution. The second is coverage gaps: some intervals along the difficulty axis contain no items at all, so genuine training gains inside those intervals may barely register in the total score.
The paper supplies two diagnostic metrics. DDM is the 1-Wasserstein distance, measuring how much probability mass must be moved, and how far, to turn the difficulty distribution of the benchmark into the target one. Max Gap is the widest interval between adjacent difficulty points, counting the boundaries back to 0 and 1.
The theory gives a hard bound
Theorem 2.1 states that if the performance difference between two models is L-Lipschitz, then |D_B − D_μ| ≤ L·W₁(ν_B, μ). Put plainly: as long as the gap under the target distribution exceeds L·W₁, no rank flip can occur on the leaderboard. Theorem 2.3 is blunter — the Max Gap is an exact asymptotic minimax detection boundary. If the width w of an improvement interval is no larger than G(B), then no matter how many times the benchmark is re-run (any k), the power of the test equals α, meaning the improvement is simply undetectable. Only when w > G(B) does power approach 1 as sampling grows.
Auditing 25 benchmarks
The authors estimate item difficulty with a 1PL IRT model fitted by Bayesian MCMC, covering 25 popular benchmarks across 7 domains. For non-agentic benchmarks they drew on 12 models ranging from 1.7B to 397B parameters; for agentic ones they used HAL results from 15 to 66 frontier closed-source models.
The headline counts: 10 of 25 benchmarks have DDM ≥ 0.10, 8 of 25 have Max Gap ≥ 0.10, and 9 of 25 have a Top-5 gap sum ≥ 0.30.
Split them apart and the pattern sharpens. Big benchmarks fail by skewing: ScienceQA (4241 items) has DDM 0.178, MATH-500 (500 items) 0.164, MBPP+ (378 items) 0.163, IFEval (541 items) 0.159, and MedQA (1273 items) 0.157 — dense coverage, but difficulty weights visibly far from uniform, so a leaderboard position partly reflects the preferences of the benchmark itself. Small benchmarks fail by leaving holes: AMC23 has only 40 items yet a Max Gap of 0.217 and a Top-5 sum of 0.539; AIME24, AIME25, and AIME26 have 30 items each, with Max Gaps of 0.179, 0.158, and 0.119 and Top-5 sums near 0.5; on the agentic side SWE-bench-Mini (50 items) has a Max Gap of 0.149 and CORE-Bench Hard (45 items) 0.113. The paper stresses that coverage gaps are not merely an artifact of tiny benchmarks — HumanEval and BBH-5 objects, with hundreds of items, show substantial gaps too.
BenchFill: dropping questions into the holes
For coverage gaps the paper proposes BenchFill: rather than writing questions from scratch, an LLM rewrites existing ones, with the original item serving as a local difficulty reference that preserves the format and style of the benchmark. Generated children then pass a validity check, collect model responses, and get their difficulty estimated through online IRT calibration, before being written back into a memory bank that guides the next round of generation. The result: with only three inserted items per benchmark, the sum of the top three difficulty gaps drops by 21.09%–38.88%.
The authors released gap-filled versions of AIME24, AIME25, AIME26, and AMC23 (item counts going 30→50, 30→52, 30→53, and 40→80), pushing the widest gaps from 0.179, 0.158, 0.119, and 0.217 down to 0.027, 0.030, 0.031, and 0.022. Code and the augmented datasets are open.
The clearest evidence is their SFT run: on AIME24/25/26, adding just one BenchFill item made positive gains statistically detectable in 3 out of 3 cases, while the original benchmarks and baseline augmentations detected none (0 out of 3 each).
What it means for leaderboard chasing
The paper frames measure mismatch and coverage gaps as two fundamental failure modes of benchmark evaluation, attaching a "to the best of our knowledge" qualifier to the novelty of that framing. Treat the priority claim as provisional; the method does not depend on it.
The real takeaway is this: if a benchmark is dense only in one band of difficulty, then "the model got better at the hard part" and "the model got better overall" are two different things, and the leaderboard only reports the second. A training team using that score to decide whether a run was wasted may be discarding a genuinely useful experiment. Looking at a scoreboard again, it may be worth asking one more question: what does the difficulty distribution of this benchmark look like, and who is it effectively weighting for?
So the next time a model "suddenly jumps" on some benchmark, do not rush to credit the architecture — the questions may simply have been denser in that difficulty band.