What 1.26 Million Blind Votes Actually Show
"Is ChatGPT getting dumber?" It is close to a reflex for anyone who lives in these models daily. The usual explanations are two: the vendor quietly swapped the model, or the model itself regressed. The first has real evidence behind it — GPT-4's accuracy on identifying prime versus composite numbers fell from 84% in March 2023 to 51% in June 2023 (arXiv:2307.09009). But a third explanation rarely gets examined seriously: the model did not change, the people judging it did.
A paper posted on arXiv on 10 November 2026 (Zihan Chen and Di Zhu of Stevens Institute of Technology, and Lei Nico Zheng of the University of Massachusetts Boston; arXiv:2611.14333) turns that possibility into a number, using the public Chatbot Arena battle log released by LMSYS. The raw log records 1,799,991 battles from 24 April 2023 to 14 August 2024. The paper drops every record before the "both are bad" button arrived on 29 April 2023, and keeps only pairs of frozen models — 33 dated vendor snapshots such as gpt-4-0314, 73 fixed open-weight checkpoints, and 23 aliases that could point at a moving model, which are excluded. The analysis sample ends up at 1,260,967 votes over 2,231 model pairs and 106 models. The battle log itself is a public file (clean_battle_20240814_public.json), so anyone can download it and rerun the analysis.
The same models, judged more harshly
The core result is clean: on the same pair of completely unchanged models, the share of votes saying "both are bad" rises by 1.71 percentage points per month (standard error 0.18, on a mean of 17.1%). In 2024, where the data is densest, the figure is 1.91 points per month.
The paper also offers a check that does not lean on the regression: among the 14 frozen pairs judged at least fifty times both in May–October 2023 and in February–August 2024, all fourteen rose, from 15.5% to 23.3%, a gain of 7.8 points.
What did not move is the ranking. Among decisive votes the higher-rated model wins 66.7% of the time, and in 2024 that trend is 0.00 points per month. What changed is not which answer people prefer but how often neither answer clears their bar. The paper frames this as a treadmill: the models improved fast (the average Bradley–Terry rating of the models competing each month rose from 823 in May 2023 to 1067 in August 2024, a gain of 245 Elo points; an August 2024 model drawn against a May 2023 one would be expected to score about 78%), yet platform-wide the "both are bad" share only moved from 19.4% to 20.6%. The bar rose; the felt experience stood still.
How many explanations were ruled out
"Picky users" sounds like buck-passing, so the paper works hard to exclude other causes. Across 28 robustness specifications the trend ranges from +0.97 to +2.61 points per month, and every single one is positive, statistically significant in all but one. Controlling for prompt language, code and math flags, and the seven difficulty tags still leaves +1.63. Restricting to the simplest single-turn English prompts gives +1.30. Restricting to dated OpenAI and Anthropic snapshot pairs gives +1.41; to fixed open-weight pairs +2.12; to pairs with stable answer length +1.16. Interface redesign does not explain it either — re-estimating within the periods between each page change leaves the trend positive.
The harder layer is the judges themselves. Among 24,508 judge identifiers seen over at least a month — 6% of identifiers, but 29% of the votes — the same person's verdicts harden by 1.90 points per month. Holding the same person on the same pair (11,642 votes across 3,437 identifier-pair combinations) gives +2.30. Splitting by arrival date, users who showed up in 2024 are already 1.25 points per month harsher on their very first vote than those who arrived in 2023, a gap reaching 11.6 points by August.
There is also a neat closing move: 17 of the 18 frequently judged frozen models show a significant rise on their own pairs, averaging +1.40 — but once month effects estimated from other models' pairs are subtracted, the average trend collapses to +0.19. The drift is shared across the platform; it is not any single model degrading.
What it means for leaderboards and human feedback
The advice to evaluation platforms and model providers points the same way: absolute and relative judgments have to be accounted for separately. The absolute judgment — is this answer good enough — drifted at an annualized rate of about 21 percentage points, while the relative judgment, picking one of two, showed no trend in 2024. A sample labelled "acceptable" in 2023 is not necessarily still acceptable under the same standard in 2024.
For leaderboards the fix is to keep a set of frozen reference models inside the evaluation stream and use same-period references to re-anchor absolute scores; the paper's out-of-sample test shows this does remove the drift. For teams doing RLHF the implication is blunter: pooling absolute labels collected in different periods mixes standards, not just data.
Do not overread it
The paper is candid about its own edges. The audience is self-selected, the judge identifier follows a device or network rather than a person, the log contains no prompt or answer text at all, and the window ends in August 2024. And the data cannot separate a higher acceptance threshold from closer scrutiny of the same answer.
But the direction of the conclusion is not weak. When a product stands still while the category around it keeps moving, it gets judged as declining on its own. Vendors reading user scores as a regression signal may be reading standard drift instead; training data mixing absolute labels from different eras mixes standards rather than measuring quality.
So the next time you think something has gotten dumber, it may be worth asking: did it change, or did you?