[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-benchleaking-base-model-contamination":3,"topics-all":38,"news-related-cf8f9537-6d2a-442b-a5c2-8096bbc7325e":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"cf8f9537-6d2a-442b-a5c2-8096bbc7325e","96%考题被默写:基准污染渗入基座模型","GAIR团队实测四个开源基座模型:ERNIE-5.5-Flash-Base能默写出96.3%的HumanEval考题与参考实现,85.4%的OpenBookQA连答案一起泄漏;把泄漏题清零后,其20基准均分从85.17掉到68.55,排序反转。论文判断基准污染已从后训练渗入预训练。","11月16日挂上arXiv的一篇论文,把「模型背题」从猜测变成了复现实验:回答完一道HumanEval编程题之后,某个开源基座模型没有停,而是接着把第67到70题按顺序默写了出来——题干、参考实现一字不差,连原题里的拼写错误examble都原样保留。论文作者来自Xinxu AI与GAIR,他们给这种现象起了个名字:benchleaking。\n\n按论文的定义,benchleaking指基准测试的考题本身进入了训练数据,分数反映的是记忆力而不是能力;它和benchmaxxing不同——后者只是朝着基准优化,考题本身没进训练集。用考试类比:一个是刷遍了同款模拟题,一个是把真题卷塞进了教材。\n\n## 默写整张考卷\n\n方法不复杂:少样本提示后让模型作答,关掉所有停止条件,让它一直写下去,再用严格匹配核对——题干、选项、参考答案必须同时出现才算泄漏,复述个大概不算。在20个基准、四个公开基座模型上跑下来,分化清晰:百度ERNIE-5.5-Flash-Base有96.3%的HumanEval题目连同参考实现被默写,OpenBookQA为85.4%,GSM8K为80.1%;小米MiMo-V2.5-Base的HumanEval泄漏也有73.8%。对照组里,DeepSeek-V4-Flash-Base与腾讯Hy3-preview-Base的HumanEval泄漏只有9.1%和6.7%,OpenBookQA与GPQA-Main的完整泄漏为零。另一条输出里,ERNIE答完MMLU抽象代数的第一道测试题后,把接下来的三道按原顺序默写了出来——题目、选项、正确答案一个不少。\n\n## 清零之后,排序反转\n\n论文定义了Anti-Hack分:查出的泄漏题全部记零分,分母不变。四个模型的20基准均分,ERNIE从85.17掉到68.55,跌16.6分——在这张四模型对比里从最高变成最低;MiMo跌5.0分,HumanEval单项从98.78掉到25.76;DeepSeek与Hy3只跌1.5和1.3分,基本没伤筋动骨。单项更夸张:ERNIE的HumanEval从100.00掉到3.66,OpenBookQA从98.60掉到14.20,GSM8K从91.21掉到17.21。论文由此判断:强基座成绩并不依赖背题,但背题足以改写排序。\n\n## 后训练早有前兆\n\n预训练之外,论文还复现并转述了两份第三方报告。一份审计发现Kimi K3.1在23.3%的尝试里说出了任务说明里根本没有、只存在于Terminal-Bench隐藏测试中的评分规则,前代Kimi K3一次也没有;把任务文件改到与该规则矛盾,K3.1依然沿用旧规则。一个音乐和声任务里,它直接报出了隐藏测试检查的四个声部音域,而这些音域从未出现在任务说明里。论文自己的复测中,K3.1 Pro在Terminal-Bench 4.0上正常解题时有14.3%的任务出现隐藏规则复述(9\u002F63),预填推理时达到51.5%(34\u002F66)。另一份MaxxBench把89道DeepSWE任务稍作修改——比如要求文件名小写——DeepSeek-V4.1-Flash的通过率就从75.8%掉到62.5%,Kimi K3.1 Pro从83.8%掉到68.0%。\n\n## 该泼的冷水\n\n几个限定必须说清:这是11月16日提交的arXiv v1预印本,未经同行评审;四模型泄漏率与Anti-Hack全部出自这一篇论文的单一实测;论文自己剔除了WinoGrande的双生句假阳性——四个模型都能「默写」12.6%到14.4%,但那是题型构造使然,作者没把它算成泄漏,这个分寸值得肯定;另外,论文脚注给出的GitHub仓库地址,截至发稿访问仍是404。论文本体:https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.21315\n\n## 所以呢\n\n基座模型是下游研究的地基:微调、蒸馏、agent评测,都默认起点干净。这篇论文说的是,这个默认不再成立;论文自己的呼吁,是把去污染报告和污染校正分变成发布标配。以后看到新的跑分,先问一句:有没有去污染报告?分数会说谎,但默写不会——当模型能一字不差背出考卷,榜单上的名次就只是记忆力的排名。\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.21315","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"a8002d98-9df1-4ab9-94d4-a7625af634c4","china-ai",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"4dd7eb41-f0c6-4d70-83cd-961e60f0ae0a","en","Benchleaking: Base Models Recite 96% of HumanEval","GAIR's audit: ERNIE-5.5-Flash-Base recites 96.3% of HumanEval; zeroing leaked items drops its 20-benchmark average from 85.17 to 68.55.","A paper posted to arXiv on November 16 turned 'models memorizing benchmarks' from suspicion into a reproducible experiment. After answering one HumanEval problem, an open-weight base model did not stop: it kept going and wrote out problems 67 through 70 of the same test set, in order, complete with task descriptions and reference implementations — down to the typo examble preserved from the original prompt. The authors, from Xinxu AI and GAIR, gave the phenomenon a name: benchleaking.\n\nBy their definition, benchleaking means a benchmark's own test examples entered the training data, so a score reflects recall rather than capability. It differs from benchmaxxing, where a developer optimizes toward a benchmark without its test set entering training. As an exam analogy: one is drilling endless look-alike mock exams; the other is stuffing the actual answer key into the textbook.\n\n## Reciting the Whole Test Set\n\nThe method is simple. Give the base model a few-shot prompt, let it answer, disable all stopping conditions, and let generation run on. A strict matching rule then decides what counts: the complete question, its options or context, and the reference answer must appear together in one generated item — paraphrases do not count. Across 20 benchmarks and four public base models, the split is stark. For Baidu's ERNIE-5.5-Flash-Base, 96.3% of HumanEval problems leak with their reference implementations, along with 85.4% of OpenBookQA and 80.1% of GSM8K. Xiaomi's MiMo-V2.5-Base leaks 73.8% of HumanEval. In the control group, DeepSeek-V4-Flash-Base and Tencent's Hy3-preview-Base leak 9.1% and 6.7% of HumanEval respectively, and neither leaks any complete OpenBookQA or GPQA-Main example. In another output, after answering the first MMLU abstract-algebra test question, ERNIE recited the next three in order — questions, options, and correct answers included.\n\n## Zero the Leaked Items and the Ranking Reverses\n\nThe paper defines an Anti-Hack score: every detected leaked example is marked wrong, and the denominator stays unchanged. On the 20-benchmark average, ERNIE falls from 85.17 to 68.55 — a 16.6-point drop that takes it from the highest of the four models to the lowest in the paper's comparison. MiMo loses 5.0 points, with its HumanEval maximum falling from 98.78 to 25.76. DeepSeek and Hy3 lose only 1.5 and 1.3 points, barely scratched. The single-benchmark falls are more dramatic: ERNIE's HumanEval goes from 100.00 to 3.66, OpenBookQA from 98.60 to 14.20, GSM8K from 91.21 to 17.21. The paper's reading: strong base-model results do not depend on memorized tests, but memorization is enough to rewrite the ranking.\n\n## Post-Training Already Showed the Signs\n\nBeyond pre-training, the paper also replicates and relays two third-party reports. One audit found that Kimi K3.1 states grading rules in 23.3% of its attempts that exist nowhere in the task instructions — only in Terminal-Bench's hidden tests — while its predecessor Kimi K3 never does; when a task file is edited to contradict such a rule, K3.1 keeps applying the old one. In one music-harmony task it recited all four vocal ranges the hidden test checks, none of which appeared in the instructions. In the authors' own replication on Terminal-Bench 4.0, K3.1 Pro produced hidden-rule candidates on 14.3% of tasks during normal solving (9\u002F63) and 51.5% under prefilled reasoning (34\u002F66). The other report, MaxxBench, slightly modifies 89 DeepSWE tasks — requiring lowercase filenames, for example — and pooled pass@1 falls from 75.8% to 62.5% for DeepSeek-V4.1-Flash and from 83.8% to 68.0% for Kimi K3.1 Pro.\n\n## The Cold Water\n\nSeveral caveats matter. This is an arXiv v1 preprint submitted November 16, not peer reviewed. The leakage rates and Anti-Hack numbers all come from this single paper's own measurements. The authors themselves excluded WinoGrande matches as false positives — all four models 'recite' 12.6% to 14.4% of it, but that follows from its twin-sentence construction, and the paper does not count it as leakage; that discipline deserves credit. And the GitHub repository linked in the paper's footnote still returns 404 as of this writing. The paper itself: https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.21315\n\n## So What\n\nBase models are the foundation of downstream research: fine-tuning, distillation, and agent evaluation all assume a clean starting point. This paper says that assumption no longer holds, and its own call is for decontamination reports and contamination-adjusted scores to become standard practice. The next time a new score lands, ask one question first: where is the decontamination report? Scores can lie; verbatim recitation cannot — when a model can write out the exam paper word for word, the leaderboard rank is a ranking of memory.\n","benchleaking-base-model-contamination","2026-11-17T13:45:00Z","2026-11-17T13:25:23.422903Z","2026-11-17T13:25:23.422912Z",true,"agent",52,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"ad3be632-49a1-44f5-9816-62c168e56467","全球大模型调用量榜前五全是\"中国造\":开源 MoE 正在重写 OpenRouter 的地理坐标","openrouter-top5-china-moe-open-source-2026w31","2026-08-02T03:30:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"37c8dc68-276b-4905-baed-1e120e86287c","小米MiMo-V2.5登顶OpenRouter周榜月榜：国产开源MoE拿下全球调用量第一","mimo-v2-5-openrouter","2026-07-28T03:00:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"dfa4fd62-754c-4c08-b7c4-5f8bc4c7f678","上线一周调用量增 68 倍,腾讯混元 Hy3 在 OpenRouter 全球登顶","tencent-hunyuan-hy3-openrouter","2026-07-16T00:01:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"4cdb4267-a0c3-423d-8005-504778932ae9","40份技术报告审计:73个主张只有19个有对照","llm-technical-report-ablation-audit","2026-11-17T05:30:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"d1d54032-073b-433e-9430-547c6ffb4a49","贵的模型，没写出更快的代码：9455 份实测","perfarena-llm-code-performance-ranking","2026-11-16T19:13:41+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"2e7a2be3-762e-46eb-b582-af299c4ec173","模型越新，拒绝越难拆？76页审计：并不均匀","refusal-abliteration-generation-audit","2026-11-16T17:45:00+00:00"]