A minimal-pair benchmark puts two sentences side by side that differ in exactly one grammatical respect, and asks a model to give the more acceptable one a higher score. The method spread from English to Russian, Quebec French and more than a hundred other languages, with BLiMP as the English dataset tested here. Its weak point is the label. Templates and generators produce pairs whose intended contrast native speakers do not share, and acceptability is graded, so speakers disagree — and benchmark authors end up paying humans to adjudicate every pair.

LLMs look like the obvious cheap substitute. A paper submitted to arXiv on 10 November turns this into an experiment: what actually happens when you let a large language model do the labelling? The author picks the three minimal-pair datasets that release every individual human judgment — English BLiMP (348 pairs), Quebec French QFrBLiMP (1693 pairs) and a Dutch set added later. Three API models from different developers (Gemini 3.8 Flash, Claude Sonnet 4.6 and GPT-6 Luna, at temperature 0) validate every pair in three ways: forced choice, a standalone rating of each sentence, and a three-model panel, each scored against panels of one, three and five human judges. The design runs in two pre-registered phases, both locked before any output existed.

High agreement, poor detection

On BLiMP, forced-choice validators agree with the held-out human majority on 0.937–0.953 of pairs — above the five-judge panel's 0.934. Yet they recover only 0.09–0.20 of the pairs whose intended label natives reject (LNS), where a single judge recovers 0.47. The LLM panel flags 1.7% of BLiMP pairs, although 5.5% are LNS. Rating each sentence on its own recalls more, but flags 5.4–23.2% of the pairs that at least three quarters of judges support.

By F1, six of seven validators fall below a single judge on BLiMP; the best, Gemini 3.8 Flash RATE, reaches 0.254 — one judge, but not three. On QFrBLiMP the best validator is the LLM panel at 0.366, also one judge and not three. The pre-registered verdict is a weak pass: the best validator's F1 minus the three-judge panel's is −0.077, with interval [−0.304, +0.150].

The errors are not random

So far this is only "LLMs are worse judges". The more damaging layer comes next: if a validator's mistakes point in the same direction as the models' mistakes, it is not filtering noise — it is selecting items on the outcome it is supposed to measure.

The paper scores every pair with 10 open models from 1.5B to 9B across seven developers, following Warstadt et al. in counting a pair correct when the intended sentence has the higher total log-probability, and restricting to pairs humans support. Human-supported pairs that the pre-registered LLM filter flags are 18.9 points harder for the models than those it accepts (95% interval [−22.1, −15.8]), negative for all 10 models. The other six validators show the same gap, from −18.2 to −32.1 points, with Holm-adjusted p < 0.001. Every one of the 70 LM-validator combinations is negative. The gap survives per-dataset splits (−15.7 on BLiMP, −19.1 on QFrBLiMP), a strongly-supported subset (−16.4), length-normalised log-probabilities (−14.5), and leaving out any single phenomenon (−20.3 to −17.5).

Is it the models or the humans?

The paper runs the obvious control: if any reviewer's disagreement marks pairs as hard, then "the validator picked out hard pairs" is not news. Human disagreement does mark hard pairs — 13.7 points for one judge, 21.7 for three. But LLM flags mark harder ones, for all seven validators, by 5.2 to 18.5 points more than a single judge, every interval below zero. Rating validators flag roughly twice as many pairs as a single judge (19.4–24.6% against 11.4%), which should normally dilute the gap; it does not. Forced-choice validators and the panel flag about as many pairs as three judges (4.3–6.2% against 5.9%) yet still show a larger gap, by 5.8 to 10.5 points.

Deleting them makes the scores look better

The cost is quantifiable. Removing the flagged pairs raises model accuracy by 5.6 points; removing the same number of random pairs changes it by 0.0. On QFrBLiMP the LLM filter removes 426 pairs, of which only 85 are genuinely LNS, and keeps 27 of the 112 LNS pairs. Every model scores higher on the LLM-filtered QFrBLiMP than on the human-filtered one. The rankings barely move — the order of the 10 models is stable across the three versions, with at most 2 of 45 model pairs swapping. The harm is to absolute scores and to which failures stay visible, not to the leaderboard.

Replication in Dutch

After phase 1, the author found a third dataset with per-rater data, BLiMP-NL: 756 pairs from 84 paradigms, rated by native Dutch speakers on a 7-point scale, giving 38 LNS pairs. The protocol was frozen before any validator or model saw the data — same three models, prompts translated sentence by sentence, the primary filter not re-selected, the same estimands and criteria. Flagged pairs are 13.4 points harder than accepted ones (interval [−21.1, −5.4]), negative for all 10 models, and removing them beats random removal by 1.6 points. Of the 70 combinations, 66 are negative.

The Dutch round also exposes where the argument is softer: on strongly supported pairs the gap is −14.8 but the interval [−30.0, +0.8] crosses zero; the human-flag control weakens to 8.5 points; and LLM-filtered and human-filtered versions give the same mean accuracy. Detection is also weaker — forced-choice validators recall 0.15–0.43 of LNS pairs against 0.49 for one rater, and the best validator reaches an F1 of 0.388 against 0.430 for three raters.

The paper's four recommendations

Do not drop LLM-flagged pairs unchecked — send them to human judges, because many are valid and they are the hard ones. Do not read LLM agreement as validation; report recall on the pairs humans reject. Use several human judges: by point estimate, a panel of three beats every LLM validator on both datasets. And report how the labels were validated, releasing per-judge data.

The limitations are stated plainly. Only three datasets exist with per-judge releases, and BLiMP's validated subset has just 19 LNS pairs, so the phase-0 verdict is a weak pass. LNS is not the same as invalidity. The validators are three API models, three prompt formats and one temperature. The models tested top out at 9B, and DeepSeek-7B is near chance on QFrBLiMP. The mechanism is correlational: the paper shows that LLM flags and model errors coincide on human-supported pairs, not why.

Where the leak is

The contribution is not "LLMs label badly". It is that the paper puts a price on a practice spreading everywhere — using large models to validate benchmarks for other large models. When validator and evaluated model draw on the same training corpus, they fail together on the same sentences, and those are precisely the sentences that separate models.

Eighteen and a half points, translated: a benchmark filtered by models systematically flatters those models, by roughly the size of the gap you were trying to measure. And the bias runs one way. Random noise cancels; self-serving bias does not — it keeps selecting toward the items the model already gets right. So the more accurate the filter, the softer the benchmark.

The paper's own answer is not "never use LLMs" but "do not let an LLM decide what gets deleted": use it as a pre-screen and route flagged items to humans, rather than treating it as the arbiter. The harder question sits outside that recommendation — when the validator and the evaluated share the same training preferences, who arbitrates the arbitrator?

The full paper is at arXiv:2611.13788.