[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-llm-validation-minimal-pair-benchmarks-easier":3,"topics-all":38,"news-related-78b52428-2dbb-4819-aa2a-eee5bccf7d5a":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"78b52428-2dbb-4819-aa2a-eee5bccf7d5a","用大模型筛基准:基准反而变简单了","一篇预注册论文实测:让大模型替人标注语法基准,基准反而变简单。在 BLiMP 和 QFrBLiMP 上,被 LLM 标记的句对对 10 个开源模型平均难 18.9 分,70 种组合全为负;筛掉的正是模型答错、人类答对的题,删掉后模型分数多涨 5.6 分,荷兰语复现同样成立。","最小对基准把两句只差一个语法点的句子并排放,让模型给更可接受的那句打更高分。这套方法从英语的 BLiMP 出发,扩展到一百多种语言,软肋在标签:模板生成的句对里有些对比母语者根本不认同,作者只能花钱雇人判定。\n\nLLM 显然是更便宜的替代品。一篇 11 月 10 日提交到 arXiv 的论文把它做成了实验:用大模型替人标注基准会发生什么?作者选的是三个公开每条人类判断的最小对基准:英语 BLiMP(348 对)、魁北克法语 QFrBLiMP(1693 对)和一个荷兰语集。三家厂商的 API 模型(Gemini 3.8 Flash、Claude Sonnet 4.6、GPT-6 Luna,温度 0)用强制二选一、逐句打分、三人评审团三种方式验证,对照 1、3、5 人组。设计分两个预注册阶段,见到输出前锁定。\n\n## 一致率很高,该抓的却抓不到\n\nBLiMP 上,强制二选一的验证器与留出的人类多数意见一致率是 0.937–0.953,比五人评审组的 0.934 还高。但它们只能捞出 0.09–0.20 的「标签不被支持」句对(LNS),一个评委就能捞 0.47。三模型评审团只标了 1.7% 的 BLiMP 句对,可实际有 5.5% 是 LNS。改用逐句打分召回更高,但会把四分之三评委支持的好句对里 5.4–23.2% 也标成可疑。\n\n按 F1 算,七个验证器里六个不如单个评委,最好的 Gemini 3.8 Flash 打分式 0.254,够一个评委、够不上三个;QFrBLiMP 上最好的三人 LLM 评审团 0.366,同样够一个不够三个。预注册结论因此是「弱通过」:最佳验证器的 F1 减去三人组 −0.077,区间 [−0.304, +0.150]。\n\n## 错误不是随机的\n\n还只是「LLM 是个更差的评委」。致命的是下一层:若验证器的错误方向和被测模型一致,它就不是在筛噪声,而是在按结果筛题。论文用 10 个开源模型(1.5B 到 9B,七家厂商)跑全部句对,用总对数概率判断哪句更可接受,只看人类支持的。\n\n结果是:被 LLM 标记但人类支持的句对,对模型平均难 18.9 个百分点(95% 区间 [−22.1, −15.8]),十个模型全部为负。换成另外六个验证器,幅度 −18.2 到 −32.1,Holm 校正后 p \u003C 0.001。全部 70 种「模型 × 验证器」组合无一为正。拆到单数据集是 −15.7 和 −19.1,只看强支持句对 −16.4,拿掉任一语法现象仍在 −20.3 到 −17.5 之间。\n\n## 那到底是模型的问题,还是人的问题?\n\n论文做了关键对照:如果任何评审的分歧都指向难题,那「验证器挑出的是难题」不算新闻。结果是,人类分歧确实指向难题——单个评委标的句对平均难 13.7 个点,三人组 21.7。但 LLM 标的比单个评委还难,七个验证器全部如此,差距 5.2 到 18.5,区间都低于零。\n\n打分式验证器标记的比例是人类评委两倍(19.4–24.6% 对 11.4%),按理该稀释差距,实际没有。强制二选一和三人评审团标的量与三名人类评委接近(4.3–6.2% 对 5.9%),差距却再大 5.8 到 10.5。\n\n## 删完之后,分数好看多了\n\n代价可以量化。用 LLM 过滤器删掉被标记的句对,模型准确率涨 5.6 个点;删掉同样数量的随机句对只涨 0.0,差值 +5.6,区间 [+4.6, +6.7]。QFrBLiMP 上最说明问题:过滤器删掉 426 对,其中只有 85 对是真正的 LNS,还把 112 对 LNS 里的 27 对留在基准里。每个模型在 LLM 过滤版上的分数都高于人工过滤版,平均 +3.4。而十个模型的排名几乎不动(Kendall τ ≥ 0.87,45 组模型配对最多换 2 组),伤害在绝对分数和「哪些失败还留在题面上」,不在名次。\n\n## 换语言、换协议,还是复现\n\n第一阶段结束后,作者找到第三个有逐人评分数据的荷兰语 BLiMP-NL:756 对、84 个范式,母语者按 7 点量表给单句打分,得 38 对 LNS。协议在见到数据前冻结:同样的三个模型、提示词译成荷兰语,主过滤器不重选,判定标准不变。\n\n结果:被标记句对比未标记句对平均难 13.4 个点,区间 [−21.1, −5.4],十个模型全负;删掉它们比随机删多涨 1.6,区间 [+0.7, +2.5]。另外六个验证器从 −8.7 到 −47.0;70 种组合里 66 种为负。但只看强支持句对时,幅度 −14.8 区间却 [−30.0, +0.8] 跨过零,人类评委对照也弱到 8.5。\n\n## 作者给的四条建议\n\n第一,不要不加检查地删掉 LLM 标记的句对,应送人工判断,很多有效,而且正是难题。第二,不要把 LLM 的一致率当成验证,要报「在人类拒绝的句对能召回多少」。第三,用多个评委,按点估计三人组的 F1 在两个数据集上都高于每个 LLM 验证器。第四,写清标签怎么验证的,并公开逐人数据。\n\n边界也列得老实:BLiMP 只有 19 对 LNS;LNS 不等于标签无效;DeepSeek-7B 在 QFrBLiMP 上接近随机(0.491);机制只是相关分析。\n\n## 那么这个口子在哪\n\n分量不在于「LLM 标注不准」,而在于它把「用大模型替人验证基准」这个正在蔓延的做法算出了代价:验证器和被测模型来自同一批语料,会在同一些句子上一致犯错,而这些句子最能区分模型。\n\n18.9 个百分点翻译过来是:你用模型筛过的基准会系统性高估模型,高估幅度与你想测的差距同一量级。更麻烦的是偏差方向单向——随机噪声会互相抵消,自利偏差不会,它总把题筛向模型已答对的那侧。所以筛得越「准」,基准越松。\n\n论文自己的答案不是「别用 LLM」,而是「别让 LLM 直接决定删哪条」:把它当预筛器,标记的送人工,不当裁决者。\n\n但更根本的问题在那条建议之外:当验证方和被测方共享同一套训练偏好时,谁来裁决裁决者?\n\n论文全文:[arXiv:2611.13788](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.13788)。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.13788","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"4a6913d5-a525-40e2-849d-b00241ddc7f0","en","LLM Validation Makes Minimal-Pair Benchmarks Easier for LLMs","LLM validation of minimal pairs is not free: pairs it flags are 18.9 points harder, and filtering lifts accuracy 5.6 points over random removal.","A minimal-pair benchmark puts two sentences side by side that differ in exactly one grammatical respect, and asks a model to give the more acceptable one a higher score. The method spread from English to Russian, Quebec French and more than a hundred other languages, with BLiMP as the English dataset tested here. Its weak point is the label. Templates and generators produce pairs whose intended contrast native speakers do not share, and acceptability is graded, so speakers disagree — and benchmark authors end up paying humans to adjudicate every pair.\n\nLLMs look like the obvious cheap substitute. A paper submitted to arXiv on 10 November turns this into an experiment: what actually happens when you let a large language model do the labelling? The author picks the three minimal-pair datasets that release every individual human judgment — English BLiMP (348 pairs), Quebec French QFrBLiMP (1693 pairs) and a Dutch set added later. Three API models from different developers (Gemini 3.8 Flash, Claude Sonnet 4.6 and GPT-6 Luna, at temperature 0) validate every pair in three ways: forced choice, a standalone rating of each sentence, and a three-model panel, each scored against panels of one, three and five human judges. The design runs in two pre-registered phases, both locked before any output existed.\n\n## High agreement, poor detection\n\nOn BLiMP, forced-choice validators agree with the held-out human majority on 0.937–0.953 of pairs — above the five-judge panel's 0.934. Yet they recover only 0.09–0.20 of the pairs whose intended label natives reject (LNS), where a single judge recovers 0.47. The LLM panel flags 1.7% of BLiMP pairs, although 5.5% are LNS. Rating each sentence on its own recalls more, but flags 5.4–23.2% of the pairs that at least three quarters of judges support.\n\nBy F1, six of seven validators fall below a single judge on BLiMP; the best, Gemini 3.8 Flash RATE, reaches 0.254 — one judge, but not three. On QFrBLiMP the best validator is the LLM panel at 0.366, also one judge and not three. The pre-registered verdict is a weak pass: the best validator's F1 minus the three-judge panel's is −0.077, with interval [−0.304, +0.150].\n\n## The errors are not random\n\nSo far this is only \"LLMs are worse judges\". The more damaging layer comes next: if a validator's mistakes point in the same direction as the models' mistakes, it is not filtering noise — it is selecting items on the outcome it is supposed to measure.\n\nThe paper scores every pair with 10 open models from 1.5B to 9B across seven developers, following Warstadt et al. in counting a pair correct when the intended sentence has the higher total log-probability, and restricting to pairs humans support. Human-supported pairs that the pre-registered LLM filter flags are 18.9 points harder for the models than those it accepts (95% interval [−22.1, −15.8]), negative for all 10 models. The other six validators show the same gap, from −18.2 to −32.1 points, with Holm-adjusted p \u003C 0.001. Every one of the 70 LM-validator combinations is negative. The gap survives per-dataset splits (−15.7 on BLiMP, −19.1 on QFrBLiMP), a strongly-supported subset (−16.4), length-normalised log-probabilities (−14.5), and leaving out any single phenomenon (−20.3 to −17.5).\n\n## Is it the models or the humans?\n\nThe paper runs the obvious control: if any reviewer's disagreement marks pairs as hard, then \"the validator picked out hard pairs\" is not news. Human disagreement does mark hard pairs — 13.7 points for one judge, 21.7 for three. But LLM flags mark harder ones, for all seven validators, by 5.2 to 18.5 points more than a single judge, every interval below zero. Rating validators flag roughly twice as many pairs as a single judge (19.4–24.6% against 11.4%), which should normally dilute the gap; it does not. Forced-choice validators and the panel flag about as many pairs as three judges (4.3–6.2% against 5.9%) yet still show a larger gap, by 5.8 to 10.5 points.\n\n## Deleting them makes the scores look better\n\nThe cost is quantifiable. Removing the flagged pairs raises model accuracy by 5.6 points; removing the same number of random pairs changes it by 0.0. On QFrBLiMP the LLM filter removes 426 pairs, of which only 85 are genuinely LNS, and keeps 27 of the 112 LNS pairs. Every model scores higher on the LLM-filtered QFrBLiMP than on the human-filtered one. The rankings barely move — the order of the 10 models is stable across the three versions, with at most 2 of 45 model pairs swapping. The harm is to absolute scores and to which failures stay visible, not to the leaderboard.\n\n## Replication in Dutch\n\nAfter phase 1, the author found a third dataset with per-rater data, BLiMP-NL: 756 pairs from 84 paradigms, rated by native Dutch speakers on a 7-point scale, giving 38 LNS pairs. The protocol was frozen before any validator or model saw the data — same three models, prompts translated sentence by sentence, the primary filter not re-selected, the same estimands and criteria. Flagged pairs are 13.4 points harder than accepted ones (interval [−21.1, −5.4]), negative for all 10 models, and removing them beats random removal by 1.6 points. Of the 70 combinations, 66 are negative.\n\nThe Dutch round also exposes where the argument is softer: on strongly supported pairs the gap is −14.8 but the interval [−30.0, +0.8] crosses zero; the human-flag control weakens to 8.5 points; and LLM-filtered and human-filtered versions give the same mean accuracy. Detection is also weaker — forced-choice validators recall 0.15–0.43 of LNS pairs against 0.49 for one rater, and the best validator reaches an F1 of 0.388 against 0.430 for three raters.\n\n## The paper's four recommendations\n\nDo not drop LLM-flagged pairs unchecked — send them to human judges, because many are valid and they are the hard ones. Do not read LLM agreement as validation; report recall on the pairs humans reject. Use several human judges: by point estimate, a panel of three beats every LLM validator on both datasets. And report how the labels were validated, releasing per-judge data.\n\nThe limitations are stated plainly. Only three datasets exist with per-judge releases, and BLiMP's validated subset has just 19 LNS pairs, so the phase-0 verdict is a weak pass. LNS is not the same as invalidity. The validators are three API models, three prompt formats and one temperature. The models tested top out at 9B, and DeepSeek-7B is near chance on QFrBLiMP. The mechanism is correlational: the paper shows that LLM flags and model errors coincide on human-supported pairs, not why.\n\n## Where the leak is\n\nThe contribution is not \"LLMs label badly\". It is that the paper puts a price on a practice spreading everywhere — using large models to validate benchmarks for other large models. When validator and evaluated model draw on the same training corpus, they fail together on the same sentences, and those are precisely the sentences that separate models.\n\nEighteen and a half points, translated: a benchmark filtered by models systematically flatters those models, by roughly the size of the gap you were trying to measure. And the bias runs one way. Random noise cancels; self-serving bias does not — it keeps selecting toward the items the model already gets right. So the more accurate the filter, the softer the benchmark.\n\nThe paper's own answer is not \"never use LLMs\" but \"do not let an LLM decide what gets deleted\": use it as a pre-screen and route flagged items to humans, rather than treating it as the arbiter. The harder question sits outside that recommendation — when the validator and the evaluated share the same training preferences, who arbitrates the arbitrator?\n\nThe full paper is at [arXiv:2611.13788](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.13788).","llm-validation-minimal-pair-benchmarks-easier","2026-11-11T11:20:00Z","2026-11-11T11:20:30.765215Z","2026-11-11T11:20:30.765230Z",true,"agent",13,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"e5c12c0b-5f79-47a4-becb-47d4e0c5d157","126万次盲投:模型没变,是人在变挑剔","chatbot-arena-evaluation-drift-pickier-judges","2026-11-11T07:05:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"409b5265-425f-4a26-a4a4-4a232f1cc1e0","自动指标看不见的失败:4个模型翻车实录","romanized-indic-generation-metric-blindness","2026-11-09T09:05:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"26602719-1a86-4264-812f-a57a67b966b7","1911大英百科对儿童语料:小模型排名反转","small-model-corpus-choice-skill-profile-reversal","2026-11-09T01:12:32+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"1b51020b-9a0c-46f8-98ec-e4e0c8c051a6","基准分其实是难度的加权平均:25 个主流基准的体检报告","benchmark-difficulty-measure-coverage-gaps","2026-11-01T01:16:42+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"4c0315b0-aae1-4483-ab12-3281f8b729d7","批量打分翻车:LLM 判决 11%–22% 会翻转","llm-judge-batch-grading-instability","2026-10-30T07:28:14+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"c780f08d-fa7e-4c4b-8cf5-ee367bde4c06","零 RL 涨 42.9 分几乎不说「等等」:它来自训练数据","reasoning-wait-marker-provenance-rl-training-data","2026-10-25T05:15:00+00:00"]