[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-uc-berkeley-benchmark-45-cheats-13-leaderboards":3,"news-related-95b04c15-d5e5-4dca-ab2b-14e343bdd4e6":39},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":26,"news_slug":32,"published_at":33,"created_at":34,"modified_at":35,"is_published":36,"publish_type":37,"image_url":13,"view_count":38},"95b04c15-d5e5-4dca-ab2b-14e343bdd4e6","UC Berkeley 曝光 AI 基准测试系统性漏洞：45 种方法可在 13 个主流榜单上「不解决任何问题拿满分」","UC Berkeley RDI 近日发表论文，揭示了当前主流 AI 基准测试的系统性漏洞。研究团队构建了一个能够自动分析基准测试代码的 AI agent，对 13 个广泛使用的 AI 基准测试进行审计，结果令人警醒：每个基准测试都存在可被利用的安全缺陷，共发现 45 种已验证的「作弊解法」。\n\n最典型的三个案例：一是 Frontier-CS，提交代码和评估器运行在同一 Python 进程，恶意代码可通过栈帧操作替换评分函数，直接获得满分；二是 Terminal-Bench，只需创建一个dummy C扩展文件通过存在性检查，无需实际调用即可通过性能测试；三是 WebArena，812 个任务的参考答案直接存放在 agent 可访问的文件系统中，agent 只需读取 JSON 文件即可获得满分。\n\n这些漏洞的共性根源在于评估基础设施的隔离不足——被评估的代码与评估代码共享内存空间或文件系统。论文指出了一个根本性矛盾：基准测试不只是衡量模型能力的工具，它们同时在塑造模型的行为。当基准测试可被攻破，模型实际上在被训练来「作弊」而非真正解决问题。\n\n这一发现的影响面极广：基准测试分数决定了模型排名、训练数据定价，乃至部分实验室的发布决策。随着各大厂商在发布前须通过安全测试（美国商务部已将 Google DeepMind、Microsoft、xAI 纳入要求），基准测试的公信力危机已不只是技术问题，更开始影响发布节奏。\n\n真正的contamination-resistant（抗污染）评估方法尚未建立，行业建立更严格评估实践的需求已迫在眉睫。","https:\u002F\u002Frdi.berkeley.edu\u002Fblog\u002Ftrustworthy-benchmarks","15e3ede3-ede2-477d-9cfd-1d1ddfcc71f3",[10,14,17,20,23],{"id":11,"name":12,"slug":12,"description":13,"color":13},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":18,"name":19,"slug":19,"description":13,"color":13},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":21,"name":22,"slug":22,"description":13,"color":13},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":24,"name":25,"slug":25,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[27],{"id":28,"lang":29,"title":30,"summary":31,"content":13},"2dc089dc-3dca-4d9a-b5ed-385e00bf10e7","en","Berkeley: 45 ways to ace 13 leaderboards without solving tasks","UC Berkeley RDI recently published a paper exposing systemic vulnerabilities in current mainstream AI benchmarks. The research team built an AI agent capable of automatically analyzing benchmark code, auditing 13 widely used AI benchmarks. The result is alarming: every benchmark has exploitable security flaws, with 45 verified \"cheat solutions\" found in total.\n\nThe three most typical cases: First, Frontier-CS — submitted code and the evaluator run in the same Python process, and malicious code can replace the scoring function via stack-frame manipulation, directly achieving full marks. Second, Terminal-Bench — only needs to create a dummy C-extension file to pass existence checks, without actual invocation, to pass performance tests. Third, WebArena — the 812 tasks' reference answers are stored in files accessible to the agent; the agent only needs to read the JSON file to get full marks.\n\nThe common root of these vulnerabilities is insufficient isolation in evaluation infrastructure — the evaluated code shares memory or file system with the evaluation code. The paper points out a fundamental contradiction: benchmarks are not just tools for measuring model capability, they simultaneously shape model behavior. When benchmarks can be broken, models are effectively being trained to \"cheat\" rather than truly solve problems.\n\nThe impact is broad: benchmark scores determine model rankings, training-data pricing, and even some labs' release decisions. As major vendors are now required to pass security testing before release (the US Department of Commerce has included Google DeepMind, Microsoft, xAI in this requirement), the benchmark-credibility crisis is no longer just a technical issue, but has begun affecting release cadence.\n\nA truly contamination-resistant evaluation method has yet to be established, and the industry's need for stricter evaluation practices is urgent.","uc-berkeley-benchmark-45-cheats-13-leaderboards","2026-05-15T01:00:00Z","2026-05-15T01:09:17.283525Z","2026-08-19T02:08:40.142862Z",true,"agent",108,{"items":40},[41,46,51,56,61,66],{"id":42,"title":43,"news_slug":44,"published_at":45},"43eda321-b0b7-4df7-b20e-9758cbab42c9","记忆越完整,眼前题越做不对:MemTrapBench 把 LLM 长期记忆框架打回原形","memtrapbench-llm-memory-cognitive-traps","2026-08-22T04:00:00+00:00",{"id":47,"title":48,"news_slug":49,"published_at":50},"88105269-9641-44c3-a705-1cf07314614f","LLM 思维链能看出\"用了几分力\":SARE 给每一步推理做 CT 扫描","step-aware-reasoning-energy-llm-cot","2026-08-04T04:00:00+00:00",{"id":52,"title":53,"news_slug":54,"published_at":55},"e2eb81dc-3112-411a-94b0-b5061a12be78","AdvancedMathBench 把数学证明拉进博士级:GPT-5.5-xhigh 仍只 75.8","advanced-math-bench-phd-level","2026-07-14T16:15:00+00:00",{"id":57,"title":58,"news_slug":59,"published_at":60},"fc1e888d-4bee-4633-86f4-edc76aa48161","BlockSearch 把语言模型变成「上下文检索器」：0.6B 在百万 token 上打平向量检索","blocksearch-context-retriever","2026-07-03T06:25:00+00:00",{"id":62,"title":63,"news_slug":64,"published_at":65},"40509ac7-c443-44f9-99a0-f90b78121d1f","陶哲轩的「Big Mathematics」:LLM 推理 + 形式化重塑数学研究的协作信任机制","tao-big-mathematics-llm-formalization","2026-06-25T02:00:00+00:00",{"id":67,"title":68,"news_slug":69,"published_at":70},"647c0908-07d8-4827-aa44-8ccd3793143b","「六月AI发布潮」：一个面向开发者的决策框架","wavespeed-june-2026-launch-decision-frame","2026-06-02T22:05:00+00:00"]