A paper posted to arXiv on November 16 turned 'models memorizing benchmarks' from suspicion into a reproducible experiment. After answering one HumanEval problem, an open-weight base model did not stop: it kept going and wrote out problems 67 through 70 of the same test set, in order, complete with task descriptions and reference implementations — down to the typo examble preserved from the original prompt. The authors, from Xinxu AI and GAIR, gave the phenomenon a name: benchleaking.
By their definition, benchleaking means a benchmark's own test examples entered the training data, so a score reflects recall rather than capability. It differs from benchmaxxing, where a developer optimizes toward a benchmark without its test set entering training. As an exam analogy: one is drilling endless look-alike mock exams; the other is stuffing the actual answer key into the textbook.
Reciting the Whole Test Set
The method is simple. Give the base model a few-shot prompt, let it answer, disable all stopping conditions, and let generation run on. A strict matching rule then decides what counts: the complete question, its options or context, and the reference answer must appear together in one generated item — paraphrases do not count. Across 20 benchmarks and four public base models, the split is stark. For Baidu's ERNIE-5.5-Flash-Base, 96.3% of HumanEval problems leak with their reference implementations, along with 85.4% of OpenBookQA and 80.1% of GSM8K. Xiaomi's MiMo-V2.5-Base leaks 73.8% of HumanEval. In the control group, DeepSeek-V4-Flash-Base and Tencent's Hy3-preview-Base leak 9.1% and 6.7% of HumanEval respectively, and neither leaks any complete OpenBookQA or GPQA-Main example. In another output, after answering the first MMLU abstract-algebra test question, ERNIE recited the next three in order — questions, options, and correct answers included.
Zero the Leaked Items and the Ranking Reverses
The paper defines an Anti-Hack score: every detected leaked example is marked wrong, and the denominator stays unchanged. On the 20-benchmark average, ERNIE falls from 85.17 to 68.55 — a 16.6-point drop that takes it from the highest of the four models to the lowest in the paper's comparison. MiMo loses 5.0 points, with its HumanEval maximum falling from 98.78 to 25.76. DeepSeek and Hy3 lose only 1.5 and 1.3 points, barely scratched. The single-benchmark falls are more dramatic: ERNIE's HumanEval goes from 100.00 to 3.66, OpenBookQA from 98.60 to 14.20, GSM8K from 91.21 to 17.21. The paper's reading: strong base-model results do not depend on memorized tests, but memorization is enough to rewrite the ranking.
Post-Training Already Showed the Signs
Beyond pre-training, the paper also replicates and relays two third-party reports. One audit found that Kimi K3.1 states grading rules in 23.3% of its attempts that exist nowhere in the task instructions — only in Terminal-Bench's hidden tests — while its predecessor Kimi K3 never does; when a task file is edited to contradict such a rule, K3.1 keeps applying the old one. In one music-harmony task it recited all four vocal ranges the hidden test checks, none of which appeared in the instructions. In the authors' own replication on Terminal-Bench 4.0, K3.1 Pro produced hidden-rule candidates on 14.3% of tasks during normal solving (9/63) and 51.5% under prefilled reasoning (34/66). The other report, MaxxBench, slightly modifies 89 DeepSWE tasks — requiring lowercase filenames, for example — and pooled pass@1 falls from 75.8% to 62.5% for DeepSeek-V4.1-Flash and from 83.8% to 68.0% for Kimi K3.1 Pro.
The Cold Water
Several caveats matter. This is an arXiv v1 preprint submitted November 16, not peer reviewed. The leakage rates and Anti-Hack numbers all come from this single paper's own measurements. The authors themselves excluded WinoGrande matches as false positives — all four models 'recite' 12.6% to 14.4% of it, but that follows from its twin-sentence construction, and the paper does not count it as leakage; that discipline deserves credit. And the GitHub repository linked in the paper's footnote still returns 404 as of this writing. The paper itself: https://arxiv.org/abs/2611.21315
So What
Base models are the foundation of downstream research: fine-tuning, distillation, and agent evaluation all assume a clean starting point. This paper says that assumption no longer holds, and its own call is for decontamination reports and contamination-adjusted scores to become standard practice. The next time a new score lands, ask one question first: where is the decontamination report? Scores can lie; verbatim recitation cannot — when a model can write out the exam paper word for word, the leaderboard rank is a ranking of memory.