[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-benchmark-difficulty-measure-coverage-gaps":3,"topics-all":38,"news-related-1b51020b-9a0c-46f8-98ec-e4e0c8c051a6":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"1b51020b-9a0c-46f8-98ec-e4e0c8c051a6","基准分其实是难度的加权平均:25 个主流基准的体检报告","NeurIPS 2026 论文把基准形式化为「难度轴上的经验测度」,指出基准存在测度错配与覆盖缺口两类失效。作者用 IRT 估计 25 个基准、7 个领域的题目难度,查出 8\u002F25 的基准最大间隔超 0.10,AIME 系列前五大间隔之和近 0.5。其 BenchFill 只需插 3 道题,就能把前三大间隔压低两成。","现在的模型评测看起来极其简单：把一套题丢给模型，跑一遍，算个平均分。问题恰恰出在「平均」这两个字上——一套题的平均分，其实是在某个难度分布上做加权平均，而这个分布从来没人检查过。\n\n一篇被 NeurIPS 2026 接收的论文把这层窗户纸捅开了：Your Benchmark Is an Empirical Measure Over Difficulty（[arXiv:2610.29927](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.29927)）。作者来自伊利诺伊大学厄巴纳-香槟分校、加州大学圣迭戈分校和纽约大学，他们把基准形式化为「难度轴上的经验测度」——n 道题各自在自己的难度 z_i ∈ [0,1] 上贡献 1\u002Fn 的质量，于是模型得分就是 ∫ s_θ(z) dν_B(z)。形式化之后，两种此前很少被命名的失效就浮出水面。\n\n## 两种失效：分布错配和覆盖空洞\n\n第一种叫测度错配（measure mismatch）：基准自身的难度分布，跟它「本来想测」的难度分布不一样，于是榜单上的结论可能和目标分布下的结论不一致。第二种叫覆盖缺口（coverage gaps）：难度轴上有些区间一道题都没有，训练在那一段取得的真实提升，在总分上可能几乎不显形。\n\n论文给了两个诊断指标：DDM 取 1-Wasserstein 距离，衡量基准难度分布与目标分布之间要搬多少质量、搬多远；Max Gap 取相邻难度点之间的最大空隙，含两端到 0 和 1 的边界。\n\n## 理论给了一个很硬的结论\n\n定理 2.1 说，若两个模型的性能差是 L-Lipschitz 的，则 |D_B − D_μ| ≤ L·W₁(ν_B, μ)；换句话说，只要目标分布下的差距超过 L·W₁，榜单排名就不可能翻转。定理 2.3 更硬：Max Gap 是渐近意义下的最优检测边界——如果一段提升的宽度 w 不超过 G(B)，那么无论把基准重复测多少次（任意 k），检验的功效都等于 α，也就是提升完全测不出来；只有 w > G(B) 时功效才随采样数趋近 1。\n\n## 25 个基准的体检结果\n\n论文用 1PL IRT 模型（贝叶斯 MCMC 拟合）估计题目难度，覆盖 7 个领域共 25 个热门基准。非 agentic 基准用了 12 个从 1.7B 到 397B 参数的模型，agentic 基准则用 15 到 66 个前沿闭源模型的 HAL 结果。\n\n统计结果有两条。论文报告：10\u002F25 的基准 DDM ≥ 0.10，8\u002F25 的 Max Gap ≥ 0.10，9\u002F25 的 Top-5 间隔之和 ≥ 0.30。\n\n分开看更有意思。**大而全的基准难在「偏」**：ScienceQA（4241 题）DDM 0.178、MATH-500（500 题）0.164、MBPP+（378 题）0.163、IFEval（541 题）0.159、MedQA（1273 题）0.157——它们题目多、覆盖密，但难度权重明显偏离均匀分布，榜单名次更多反映的是这套题自己的偏好。**小而精的基准难在「洞」**：AMC23 只有 40 题，Max Gap 0.217、Top-5 之和 0.539；AIME24\u002F25\u002F26 各 30 题，Max Gap 分别是 0.179、0.158、0.119，Top-5 之和都在 0.5 附近；agentic 一侧 SWE-bench-Mini（50 题）Max Gap 0.149、CORE-Bench Hard（45 题）0.113。论文特别指出，覆盖缺口不只是小基准的毛病——几百题的 HumanEval 和 BBH-5 objects 同样有明显空隙。\n\n## BenchFill：往洞里填题\n\n针对覆盖缺口，论文提出 BenchFill：不从零写题，而是让 LLM 改写现有题目——原题充当「局部难度参照」，保持基准原有的格式和风格；生成的子题再做有效性检查、收集模型作答、在线 IRT 标定难度，然后写回记忆库，用离目标难度最近的题做下一轮的种子。结果是每个基准只插 3 道题，前三大难度间隔之和就降了 21.09%–38.88%。\n\n作者放出了 AIME24\u002F25\u002F26 和 AMC23 的补全版本（题量 30→50、30→52、30→53、40→80），最大间隔从 0.179、0.158、0.119、0.217 压到 0.027、0.030、0.031、0.022。代码和补全后的数据集都已开源。\n\n最能说明问题的是他们的 SFT 实验：在 AIME24\u002F25\u002F26 上，只加 1 道 BenchFill 题，就让正向提升在 3\u002F3 的情况下被统计显著地检测到；而原始基准和基线增强方案都是 0\u002F3，一例都没测出来。\n\n## 这对刷榜意味着什么\n\n论文把测度错配和覆盖缺口称为基准评测的两大基本失效，并给这个优先权说法加了「据我们所知」的限定。这个声明姑且记下，但方法本身的价值不依赖它。\n\n真正的启发在最后：如果一个基准只在某一段难度上密集布题，那么「模型在难的那部分变强了」和「模型整体变强了」在总分上是两回事，而榜单只报后者。训练团队据此决定「这个 run 是不是白跑了」，可能正在丢一次有价值的实验。反过来说，以后看榜单不妨多问一句：这套题的难度分布长什么样？它到底在替谁加权？\n\n所以下次看到某个模型「某基准突然暴涨」，先别急着归因给架构——也可能只是这套题恰好在那个难度段上更密。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.29927","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"884f51f7-d1de-4aee-a298-417a577bb422","en","Benchmarks Are Weighted Difficulty Averages: 25 Audited","NeurIPS 2026: benchmarks are weighted averages over item difficulty, hiding measure mismatch and coverage gaps. Of 25, 8 show max gaps above 0.10.","Model evaluation looks deceptively simple: run a model on a set of questions, average the result. The trouble sits precisely in that word \"average\" — a benchmark score is a weighted average over some distribution of difficulty, and nobody has been checking what that distribution looks like.\n\nA paper accepted at NeurIPS 2026 tears the wrapper off: *Your Benchmark Is an Empirical Measure Over Difficulty* ([arXiv:2610.29927](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.29927)). Its authors, from the University of Illinois Urbana-Champaign, UC San Diego, and New York University, formalize a benchmark as an empirical measure over an axis of difficulty: each of n items contributes equal mass 1\u002Fn at its own difficulty z_i ∈ [0,1], so the score of a model becomes ∫ s_θ(z) dν_B(z). Once written this way, two failure modes that were rarely named before come into view.\n\n## Two failures: distributional mismatch and coverage holes\n\nThe first is *measure mismatch*: the difficulty distribution of the benchmark differs from the distribution it was meant to test, so leaderboard conclusions can diverge from conclusions under the target distribution. The second is *coverage gaps*: some intervals along the difficulty axis contain no items at all, so genuine training gains inside those intervals may barely register in the total score.\n\nThe paper supplies two diagnostic metrics. DDM is the 1-Wasserstein distance, measuring how much probability mass must be moved, and how far, to turn the difficulty distribution of the benchmark into the target one. Max Gap is the widest interval between adjacent difficulty points, counting the boundaries back to 0 and 1.\n\n## The theory gives a hard bound\n\nTheorem 2.1 states that if the performance difference between two models is L-Lipschitz, then |D_B − D_μ| ≤ L·W₁(ν_B, μ). Put plainly: as long as the gap under the target distribution exceeds L·W₁, no rank flip can occur on the leaderboard. Theorem 2.3 is blunter — the Max Gap is an exact asymptotic minimax detection boundary. If the width w of an improvement interval is no larger than G(B), then no matter how many times the benchmark is re-run (any k), the power of the test equals α, meaning the improvement is simply undetectable. Only when w > G(B) does power approach 1 as sampling grows.\n\n## Auditing 25 benchmarks\n\nThe authors estimate item difficulty with a 1PL IRT model fitted by Bayesian MCMC, covering 25 popular benchmarks across 7 domains. For non-agentic benchmarks they drew on 12 models ranging from 1.7B to 397B parameters; for agentic ones they used HAL results from 15 to 66 frontier closed-source models.\n\nThe headline counts: 10 of 25 benchmarks have DDM ≥ 0.10, 8 of 25 have Max Gap ≥ 0.10, and 9 of 25 have a Top-5 gap sum ≥ 0.30.\n\nSplit them apart and the pattern sharpens. **Big benchmarks fail by skewing**: ScienceQA (4241 items) has DDM 0.178, MATH-500 (500 items) 0.164, MBPP+ (378 items) 0.163, IFEval (541 items) 0.159, and MedQA (1273 items) 0.157 — dense coverage, but difficulty weights visibly far from uniform, so a leaderboard position partly reflects the preferences of the benchmark itself. **Small benchmarks fail by leaving holes**: AMC23 has only 40 items yet a Max Gap of 0.217 and a Top-5 sum of 0.539; AIME24, AIME25, and AIME26 have 30 items each, with Max Gaps of 0.179, 0.158, and 0.119 and Top-5 sums near 0.5; on the agentic side SWE-bench-Mini (50 items) has a Max Gap of 0.149 and CORE-Bench Hard (45 items) 0.113. The paper stresses that coverage gaps are not merely an artifact of tiny benchmarks — HumanEval and BBH-5 objects, with hundreds of items, show substantial gaps too.\n\n## BenchFill: dropping questions into the holes\n\nFor coverage gaps the paper proposes BenchFill: rather than writing questions from scratch, an LLM rewrites existing ones, with the original item serving as a local difficulty reference that preserves the format and style of the benchmark. Generated children then pass a validity check, collect model responses, and get their difficulty estimated through online IRT calibration, before being written back into a memory bank that guides the next round of generation. The result: with only three inserted items per benchmark, the sum of the top three difficulty gaps drops by 21.09%–38.88%.\n\nThe authors released gap-filled versions of AIME24, AIME25, AIME26, and AMC23 (item counts going 30→50, 30→52, 30→53, and 40→80), pushing the widest gaps from 0.179, 0.158, 0.119, and 0.217 down to 0.027, 0.030, 0.031, and 0.022. Code and the augmented datasets are open.\n\nThe clearest evidence is their SFT run: on AIME24\u002F25\u002F26, adding just one BenchFill item made positive gains statistically detectable in 3 out of 3 cases, while the original benchmarks and baseline augmentations detected none (0 out of 3 each).\n\n## What it means for leaderboard chasing\n\nThe paper frames measure mismatch and coverage gaps as two fundamental failure modes of benchmark evaluation, attaching a \"to the best of our knowledge\" qualifier to the novelty of that framing. Treat the priority claim as provisional; the method does not depend on it.\n\nThe real takeaway is this: if a benchmark is dense only in one band of difficulty, then \"the model got better at the hard part\" and \"the model got better overall\" are two different things, and the leaderboard only reports the second. A training team using that score to decide whether a run was wasted may be discarding a genuinely useful experiment. Looking at a scoreboard again, it may be worth asking one more question: what does the difficulty distribution of this benchmark look like, and who is it effectively weighting for?\n\nSo the next time a model \"suddenly jumps\" on some benchmark, do not rush to credit the architecture — the questions may simply have been denser in that difficulty band.","benchmark-difficulty-measure-coverage-gaps","2026-11-01T01:16:42Z","2026-11-01T01:17:53.494987Z","2026-11-01T01:17:53.494995Z",true,"agent",43,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"4c0315b0-aae1-4483-ab12-3281f8b729d7","批量打分翻车:LLM 判决 11%–22% 会翻转","llm-judge-batch-grading-instability","2026-10-30T07:28:14+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"c780f08d-fa7e-4c4b-8cf5-ee367bde4c06","零 RL 涨 42.9 分几乎不说「等等」:它来自训练数据","reasoning-wait-marker-provenance-rl-training-data","2026-10-25T05:15:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"c368ad9f-9308-4a0b-8f5c-3ae4601b48b9","D2K-Bench: 专家设计把 LLM 写 GPU 核提速 33.9%","d2k-bench-llm-gpu-kernel-design-guidance","2026-10-07T03:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"43eda321-b0b7-4df7-b20e-9758cbab42c9","记忆越完整,眼前题越做不对:MemTrapBench 把 LLM 长期记忆框架打回原形","memtrapbench-llm-memory-cognitive-traps","2026-08-22T04:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"88105269-9641-44c3-a705-1cf07314614f","LLM 思维链能看出\"用了几分力\":SARE 给每一步推理做 CT 扫描","step-aware-reasoning-energy-llm-cot","2026-08-04T04:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"95b04c15-d5e5-4dca-ab2b-14e343bdd4e6","UC Berkeley 曝光 AI 基准测试系统性漏洞：45 种方法可在 13 个主流榜单上「不解决任何问题拿满分」","uc-berkeley-benchmark-45-cheats-13-leaderboards","2026-05-15T01:00:00+00:00"]