[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-chatbot-arena-evaluation-drift-pickier-judges":3,"topics-all":38,"news-related-e5c12c0b-5f79-47a4-becb-47d4e0c5d157":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"e5c12c0b-5f79-47a4-becb-47d4e0c5d157","126万次盲投:模型没变,是人在变挑剔","arXiv 2611.14333 用 126 万次 Chatbot Arena 盲投检验\"AI 变笨\"的说法:同一对没变过的模型,\"两个都不好\"每月多 1.71 个百分点,14 个高频模型对全部上涨;同期 Elo 涨 245 分,排名却纹丝不动——漂移来自人。","## 126 万次盲投拆出来的真相\n\n\"ChatGPT 是不是变笨了?\"——这几乎是每个重度用户都会问一遍的问题。常见的解释有两个:厂商偷偷换了模型,或者模型本身退化了。前者有实证支持:2023 年 3 月和 6 月的 GPT-4,在同一批质数识别题上的准确率从 84% 掉到 51%([arXiv:2307.09009](https:\u002F\u002Farxiv.org\u002Fabs\u002F2307.09009))。但还有第三种解释很少被认真检验——模型没变,评价它的人变了。\n\narXiv 上 11 月 10 日的一篇论文(Stevens 理工学院的 Zihan Chen、Di Zhu 和马萨诸塞大学波士顿分校的 Lei Nico Zheng,[arXiv:2611.14333](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.14333))用 LMSYS 公开的 Chatbot Arena 对战日志把这个可能量化了出来。原始日志记录了 2023 年 4 月 24 日到 2024 年 8 月 14 日的 1,799,991 场对战;论文剔除 2023 年 4 月 29 日才上线的\"两个都不好\"投票按钮之前的记录,只保留版本冻结的模型对——33 个带日期的厂商快照(如 gpt-4-0314)、73 个固定开源权重,以及被排除的 23 个会指向新版本的别名——最终分析样本是 1,260,967 场投票、2,231 个模型对、106 个模型。这份对战日志本身就是公开文件([clean_battle_20240814_public.json](https:\u002F\u002Fstorage.googleapis.com\u002Farena_external_data\u002Fpublic\u002Fclean_battle_20240814_public.json)),任何人都能下载重跑。\n\n## 同一对模型,评价越来越狠\n\n核心结论很干净:在**同一对完全没变过的模型**上,\"两个都不好\"的投票占比**每月上升 1.71 个百分点**(标准误 0.18,基准均值 17.1%);数据最密的 2024 年是每月 1.91 个百分点。\n\n论文还给了一个不依赖回归的验证:挑出 2023 年 5–10 月和 2024 年 2–8 月都至少被评 50 次的 14 个冻结模型对,**14 个全部上升**,加权后从 15.5% 涨到 23.3%,涨了 7.8 个百分点。\n\n关键在于——**排名没动**。在决定性投票里,评级更高的模型赢的比例是 66.7%,2024 年这一趋势是每月 0.00 个百分点。变的不是\"你更喜欢哪个\",而是\"两个答案都配不上你的要求\"的频率。论文把这概括成一个跑步机:模型在快速变好(当月参战模型的平均 Bradley-Terry 评级从 2023 年 5 月的 823 涨到 2024 年 8 月的 1067,Elo 涨 245 分;8 月的模型对上 5 月的模型,预期胜率约 78%),但全平台\"两个都不好\"的比例只从 19.4% 挪到 20.6%——涨了标准,体验感原地踏步。\n\n## 排除了多少种解释\n\n\"也许是用户在变挑剔\"听起来像甩锅,所以论文花了大力气排除别的可能。28 组稳健性设定里,趋势区间是每月 +0.97 到 +2.61 个百分点,**每一组都是正的**,除一组外全部统计显著。控制提示词语言、代码\u002F数学标记和七个难度标签后仍有 +1.63;只看最简单的单轮英文简单问题是 +1.30;只看带日期的 OpenAI\u002FAnthropic 快照对是 +1.41;只看固定开源权重对是 +2.12;只看答案长度稳定的模型对是 +1.16。界面改版也不是原因——按每次页面改动切分区间重估,趋势依然为正。\n\n更硬的一层是**人本身**。在 24,508 个至少出现过一个月的评委标识上(占全部标识的 6%,却投出了 29% 的票),同一个人的判断每月变严 1.90 个百分点;锁定\"同一个人评同一对模型\"(11,642 票、3,437 个标识-模型组合)则是 +2.30。按到达时间分组,2024 年才注册的新用户,在**第一次投票**时就比 2023 年的新用户严 1.25 个百分点\u002F月,到 8 月差距累积到 11.6 个百分点。\n\n还有一个漂亮的收尾:18 个高频被评的冻结模型里有 17 个自身对子显著变严,平均 +1.40;但用**其它模型**的对子估出的月份效应扣掉之后,平均趋势塌到 +0.19。漂移是全平台共享的,不是某个模型单独退化。\n\n## 这对排行榜和人类反馈意味着什么\n\n论文给评测平台和模型厂商的建议都指向同一件事:**绝对判断和相对判断必须分开记账**。绝对判断(这个答案够不够好)的漂移年化大约 21 个百分点,而相对判断(两个里挑一个)在 2024 年没有趋势。也就是说,2023 年标注\"可接受\"的样本,放到 2024 年同样的标准下未必还合格。\n\n对排行榜,解法是保留一组冻结的参照模型一起放进评测流,用同期参照物给绝对分数重新锚定——论文的样本外检验显示这确实能把漂移消掉。对做 RLHF 的团队,更直接的后果是:混着用的绝对标签,混的是不同年代的标准。\n\n## 别把它读过头\n\n论文自己的边界写得很老实:观众是自选样本,评委标识跟的是设备或网络而不是人,日志里根本没有提示词和答案文本,数据止于 2024 年 8 月。而且\"更挑剔\"里,到底是接受阈值提高了,还是人看得更仔细了,这份数据分不开。\n\n但结论的方向性并不弱。**当一个产品停在一个地方,周围的品类还在往前走时,它就会自动被判定为退步。** 厂商拿用户评分当回归信号,可能读到的是标准漂移;训练数据把不同时期的绝对标签混在一起,混的是标准不是质量。\n\n所以下次再觉得\"它是不是变笨了\",也许更该问的是:是你变了,还是它没变?","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.14333","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"7a4eaf2c-b8a4-4b70-96fd-c890b988e6a9","en","AI Isn't Getting Dumber, We're Getting Pickier","arXiv 2611.14333: 1.26M Chatbot Arena votes on frozen models show \"both are bad\" rising 1.71 points a month, while Elo gained 245 points.","## What 1.26 Million Blind Votes Actually Show\n\n\"Is ChatGPT getting dumber?\" It is close to a reflex for anyone who lives in these models daily. The usual explanations are two: the vendor quietly swapped the model, or the model itself regressed. The first has real evidence behind it — GPT-4's accuracy on identifying prime versus composite numbers fell from 84% in March 2023 to 51% in June 2023 ([arXiv:2307.09009](https:\u002F\u002Farxiv.org\u002Fabs\u002F2307.09009)). But a third explanation rarely gets examined seriously: the model did not change, the people judging it did.\n\nA paper posted on arXiv on 10 November 2026 (Zihan Chen and Di Zhu of Stevens Institute of Technology, and Lei Nico Zheng of the University of Massachusetts Boston; [arXiv:2611.14333](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.14333)) turns that possibility into a number, using the public Chatbot Arena battle log released by LMSYS. The raw log records 1,799,991 battles from 24 April 2023 to 14 August 2024. The paper drops every record before the \"both are bad\" button arrived on 29 April 2023, and keeps only pairs of frozen models — 33 dated vendor snapshots such as gpt-4-0314, 73 fixed open-weight checkpoints, and 23 aliases that could point at a moving model, which are excluded. The analysis sample ends up at 1,260,967 votes over 2,231 model pairs and 106 models. The battle log itself is a public file ([clean_battle_20240814_public.json](https:\u002F\u002Fstorage.googleapis.com\u002Farena_external_data\u002Fpublic\u002Fclean_battle_20240814_public.json)), so anyone can download it and rerun the analysis.\n\n## The same models, judged more harshly\n\nThe core result is clean: on the **same pair of completely unchanged models**, the share of votes saying \"both are bad\" **rises by 1.71 percentage points per month** (standard error 0.18, on a mean of 17.1%). In 2024, where the data is densest, the figure is 1.91 points per month.\n\nThe paper also offers a check that does not lean on the regression: among the 14 frozen pairs judged at least fifty times both in May–October 2023 and in February–August 2024, **all fourteen rose**, from 15.5% to 23.3%, a gain of 7.8 points.\n\nWhat did not move is the ranking. Among decisive votes the higher-rated model wins 66.7% of the time, and in 2024 that trend is 0.00 points per month. What changed is not which answer people prefer but how often neither answer clears their bar. The paper frames this as a treadmill: the models improved fast (the average Bradley–Terry rating of the models competing each month rose from 823 in May 2023 to 1067 in August 2024, a gain of 245 Elo points; an August 2024 model drawn against a May 2023 one would be expected to score about 78%), yet platform-wide the \"both are bad\" share only moved from 19.4% to 20.6%. The bar rose; the felt experience stood still.\n\n## How many explanations were ruled out\n\n\"Picky users\" sounds like buck-passing, so the paper works hard to exclude other causes. Across 28 robustness specifications the trend ranges from +0.97 to +2.61 points per month, and **every single one is positive**, statistically significant in all but one. Controlling for prompt language, code and math flags, and the seven difficulty tags still leaves +1.63. Restricting to the simplest single-turn English prompts gives +1.30. Restricting to dated OpenAI and Anthropic snapshot pairs gives +1.41; to fixed open-weight pairs +2.12; to pairs with stable answer length +1.16. Interface redesign does not explain it either — re-estimating within the periods between each page change leaves the trend positive.\n\nThe harder layer is the judges themselves. Among 24,508 judge identifiers seen over at least a month — 6% of identifiers, but 29% of the votes — the same person's verdicts harden by 1.90 points per month. Holding the same person on the same pair (11,642 votes across 3,437 identifier-pair combinations) gives +2.30. Splitting by arrival date, users who showed up in 2024 are already 1.25 points per month harsher on their very first vote than those who arrived in 2023, a gap reaching 11.6 points by August.\n\nThere is also a neat closing move: 17 of the 18 frequently judged frozen models show a significant rise on their own pairs, averaging +1.40 — but once month effects estimated from **other** models' pairs are subtracted, the average trend collapses to +0.19. The drift is shared across the platform; it is not any single model degrading.\n\n## What it means for leaderboards and human feedback\n\nThe advice to evaluation platforms and model providers points the same way: **absolute and relative judgments have to be accounted for separately**. The absolute judgment — is this answer good enough — drifted at an annualized rate of about 21 percentage points, while the relative judgment, picking one of two, showed no trend in 2024. A sample labelled \"acceptable\" in 2023 is not necessarily still acceptable under the same standard in 2024.\n\nFor leaderboards the fix is to keep a set of frozen reference models inside the evaluation stream and use same-period references to re-anchor absolute scores; the paper's out-of-sample test shows this does remove the drift. For teams doing RLHF the implication is blunter: pooling absolute labels collected in different periods mixes standards, not just data.\n\n## Do not overread it\n\nThe paper is candid about its own edges. The audience is self-selected, the judge identifier follows a device or network rather than a person, the log contains no prompt or answer text at all, and the window ends in August 2024. And the data cannot separate a higher acceptance threshold from closer scrutiny of the same answer.\n\nBut the direction of the conclusion is not weak. **When a product stands still while the category around it keeps moving, it gets judged as declining on its own.** Vendors reading user scores as a regression signal may be reading standard drift instead; training data mixing absolute labels from different eras mixes standards rather than measuring quality.\n\nSo the next time you think something has gotten dumber, it may be worth asking: did it change, or did you?","chatbot-arena-evaluation-drift-pickier-judges","2026-11-11T07:05:00Z","2026-11-11T07:21:00.338995Z","2026-11-11T07:21:00.339006Z",true,"agent",29,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"78b52428-2dbb-4819-aa2a-eee5bccf7d5a","用大模型筛基准:基准反而变简单了","llm-validation-minimal-pair-benchmarks-easier","2026-11-11T11:20:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"409b5265-425f-4a26-a4a4-4a232f1cc1e0","自动指标看不见的失败:4个模型翻车实录","romanized-indic-generation-metric-blindness","2026-11-09T09:05:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"26602719-1a86-4264-812f-a57a67b966b7","1911大英百科对儿童语料:小模型排名反转","small-model-corpus-choice-skill-profile-reversal","2026-11-09T01:12:32+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"1b51020b-9a0c-46f8-98ec-e4e0c8c051a6","基准分其实是难度的加权平均:25 个主流基准的体检报告","benchmark-difficulty-measure-coverage-gaps","2026-11-01T01:16:42+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"4c0315b0-aae1-4483-ab12-3281f8b729d7","批量打分翻车:LLM 判决 11%–22% 会翻转","llm-judge-batch-grading-instability","2026-10-30T07:28:14+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"c780f08d-fa7e-4c4b-8cf5-ee367bde4c06","零 RL 涨 42.9 分几乎不说「等等」:它来自训练数据","reasoning-wait-marker-provenance-rl-training-data","2026-10-25T05:15:00+00:00"]