[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-bestrand-benchmark-compression-random-sampling":3,"topics-all":38,"news-related-b8ca86ba-2401-441d-ba5c-06cb6ef09420":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"b8ca86ba-2401-441d-ba5c-06cb6ef09420","随机抽样几乎就够了:高误差风险再砍50.2%","随机抽样评测终于有了理论说明书：多数子集误差小、少数大得多，长尾源自模型同质化，评再多模型也平均不掉，压缩评测也替代不了最终的全量验证。BestRand 用 5 个探针模型从 128 个随机子集里挑最均衡的一个，70 个基准实测高误差概率降 50.2%，MAE 降 16.4%，代码与数据全部开源。","评测开销正在变成一笔没人细算的账。模型发布的节奏按周计，而一个基准动辄几千上万道题：全量跑一遍 MMLU 或 OSWorld，时间和算力都吃不消。「基准压缩」因此成了刚需——从 N 道题里抽一小撮子集，用子集平均分去估全量分。但最基础的问题一直悬着：这个估计到底有多准？随手随机抽，靠谱吗？\n\n## 随机抽样为什么「几乎够用」\n\n[arXiv 2611.00883](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.00883) 这篇论文先把理论补上了。Hugging Face 论文页把团队标注在卡内基梅隆大学名下，一作 George Ling 与 Shanshan Zhong 等六位作者推导出用子集估全量分的误差分布解析近似，结论一句话：大多数子集误差都很小，少数子集误差大得多，整条分布是长尾的。理论上，随着被评模型数 M 增大，MAE 分布的峰值会以 C0\u002F√M 的通用速率（C0≈0.578）逼近分布下端点——也就是说，随机抽出来的子集大概率落在低误差区，「随机抽样几乎够用」不是修辞，是被证明了的现象。论文声称数值实验与理论预测高度一致。\n\n## 长尾的根源：模型同质化\n\n真正要命的不是平均误差，而是那条尾巴：偶尔抽到一撮整体偏难（或偏简单）的题，所有模型的分数会一起被压低（或抬高）。论文把这叫「共同偏移」，其尺度随模型间行为相关性 ρ 增大，而 ρ 的来源正是当代基础模型的同质化——相似的骨干网络、重叠的训练语料，再加上蒸馏把行为进一步对齐。最扎眼的结论是：固定 ρ>0 时，被评模型数趋于无穷，这部分方差的极限仍是 c(k)σ√ρ>0，平均不掉。换句话说，大家把模型训得越像，共享的错误越多，抽样评测的天花板就越硬。\n\n## BestRand：五个探针模型，挑掉「不走运」的子集\n\n方法朴素：先从 N 道题里随机抽 min(4k,N) 道做候选池；用 5 个探针模型只在池上跑一遍拿分；再从池里随机生成 128 个大小为 k 的候选子集；最后挑出探针均分向量与池均分向量 L2 距离最小的那个子集，目标模型只评它。探针随机选、L2 匹配是默认配置。128 个候选全部复用同一批缓存的探针分数，选择本身就是一次批量张量运算——论文的计算资源声明写着：子集选择只要 CPU，并行执行不到 0.1 秒，全部压缩实验单卡 GPU 就能跑完。理论上也有保证：论文证明当探针匹配足够准时，共同偏移方差会收敛到严格低于随机抽样的水平。相比要攒大量历史评测记录的响应式方法，它的构建开销只有一次小规模探针评估。\n\n## 数字与边界\n\n实验覆盖 70 个基准：44 个文本、16 个多模态、10 个智能体，从 MMLU、GSM8K、HumanEval 到 MMMU、MathVista、MME，再到 OSWorld、tau-bench；保留率从 2% 到 50% 共六档；对比 Random、SparseEval、Anchor Points、TinyBenchmarks、ZipBench 等六个代表基线，Random 和 BestRand 各跑 10000 个种子，其余方法只用 3 个。对 Random、ZipBench、TinyBenchmarks 三个强基线，BestRand 把 MAE 平均再降 16.4%；以「超过 Random 第 70 百分位或均值」定义高误差，高误差概率平均降 50.2%；构建记录平均只有数据依赖类基线的 51.3%（2% 保留率时低到 8.0%）；在分数段、模型家族、时间三类偏移下，OOD 估计误差平均再降 21.8%，并进一步扩展到了 VLA 评测。局限论文自己写得很清楚：压缩评测不能保证零误差，最终发布前仍建议跑全量；探针推理成本不为零，只是小。\n\n代码以 MIT 协议开源，[GitHub 仓库](https:\u002F\u002Fgithub.com\u002FGeorge-Ling3\u002FBestRand)连着约 86 MiB 的 BestRand-Data 一起放出：GPQA、BBH、TruthfulQA、HellaSwag、IFEval、MMLU-Pro、MMLU、MuSR、WinoGrande 等 12 个基准的完整 prompt 与模型-题目得分矩阵、5 个构建模型加 20 个留出测试模型的固定切分，NumPy 加 SciPy，纯 CPU 可复现。\n\n## 所以呢\n\n这篇文章顺手给「模型同质化」做了一次量化：当所有模型在同一批题上一起翻车，连统计平均都救不回来。下次看到某个「精简版」榜单，不妨先问一句：这撮题是谁挑的、拿几个模型校过？答案如果是「随便随机挑的」，现在你知道那条长尾还在等着谁。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.00883","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"2d9c2fb0-2be5-4ad1-aedb-e9747addf355","compression",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"a24a01cc-afe5-472e-a3dd-2688fffc0de9","en","Random Is Almost Enough: BestRand Cuts High-Error Risk by 50.2%","Theory: random subsets are almost enough, but homogeneity leaves a long tail. BestRand: 5 probes pick from 128 subsets, high-error risk down 50.2%.","Evaluation cost is becoming the line item nobody wants to audit. Models ship weekly, while benchmarks pack thousands of items; running MMLU or OSWorld end to end eats serious time and compute. So the industry leans on benchmark compression: draw a small subset of k items from N, and estimate the full-benchmark score from the subset average. The basic question, though, has stayed open: how accurate is that estimate — and is plain random sampling good enough?\n\n## Why random sampling is \"almost enough\"\n\nThe [paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.00883) closes the theory gap first. The Hugging Face paper page carries a Carnegie Mellon University org badge; first author George Ling, together with Shanshan Zhong and four co-authors, derives an analytical approximation to the error distribution of subset-based estimates. In one sentence: most subsets give small errors, a few give much larger ones — the distribution is long-tailed. As the number of evaluated models M grows, the mode of the MAE distribution approaches the lower endpoint at a universal rate of C0\u002F√M with C0≈0.578. Random subsets land in the low-error region with high probability; \"almost enough\" is a theorem, not a slogan. Numerical experiments, the authors report, closely track the theoretical predictions.\n\n## The tail comes from model homogeneity\n\nThe real problem is not the average error but the tail: occasionally you draw a batch of items that is systematically harder or easier, and every model's score moves together. The paper calls this the common shift; its scale grows with the behavioral correlation ρ across models — and ρ is exactly what foundation-model homogeneity produces: similar or identical backbones, overlapping training corpora, and distillation aligning behavior further. The sharpest result: for fixed ρ>0, even as the number of evaluated models goes to infinity, this variance component converges to c(k)σ√ρ>0. Averaging over more models cannot wash out shared errors. The more alike we train our models, the harder the ceiling on sampling-based evaluation.\n\n## BestRand: five probe models to dodge unlucky subsets\n\nThe method is disarmingly simple. Draw a candidate pool of min(4k,N) items; score 5 probe models on that pool only; generate 128 random candidate subsets of size k from the pool; pick the subset whose mean probe-score vector is closest in squared L2 distance to the pool mean, and evaluate target models on that subset alone. Random probe selection and L2 matching are the defaults. All 128 candidates reuse the same cached probe scores, so selection is a single batched tensor operation — the paper's computational resources statement notes that subset selection runs on CPU in under 0.1 seconds with parallel execution, and all compression experiments fit on a single GPU. There is a theoretical guarantee too: with sufficiently accurate probe matching, the common-shift variance converges to a level strictly below random sampling. Versus response-based methods that need large archives of past evaluations, the only construction overhead is one small probe pass.\n\n## Numbers and boundaries\n\nThe evaluation spans 70 benchmarks — 44 text, 16 multimodal, 10 agent — from MMLU, GSM8K, and HumanEval to MMMU, MathVista, and MME, to OSWorld and tau-bench; six retention ratios from 2% to 50%; six representative baselines including Random, SparseEval, Anchor Points, TinyBenchmarks, and ZipBench, with 10,000 seeds each for Random and BestRand versus three for the others. Against three strong baselines (Random, ZipBench, TinyBenchmarks), BestRand cuts average MAE by 16.4%; with high error defined as exceeding Random's 70th percentile or mean, it cuts high-error probability by 50.2%; construction records average 51.3% of data-dependent baselines (down to 8.0% at 2% retention); under score-range, model-family, and temporal shifts, OOD estimation error drops by another 21.8% on average, and the method extends to VLA evaluation. The limitations section stays honest: compression cannot guarantee zero estimation error, full-benchmark runs are still recommended for final evaluations, and probe inference cost is nonzero — just small.\n\nCode is open-sourced under MIT, and the [GitHub repo](https:\u002F\u002Fgithub.com\u002FGeorge-Ling3\u002FBestRand) bundles roughly 86 MiB of BestRand-Data: full prompts and model-item score matrices for 12 benchmarks including GPQA, BBH, TruthfulQA, HellaSwag, IFEval, MMLU-Pro, MMLU, MuSR, and WinoGrande; fixed splits of 5 construction models plus 20 held-out test models; reproducible with NumPy and SciPy on CPU alone.\n\n## So what\n\nThe quiet contribution here is quantifying model homogeneity: when every model fails on the same items, even statistical averaging cannot save you. Next time a \"lite\" leaderboard crosses your feed, ask who picked those items and how many models calibrated the pick. If the answer is \"just sampled randomly,\" you now know exactly which tail is waiting.","bestrand-benchmark-compression-random-sampling","2026-11-06T15:40:00Z","2026-11-06T15:12:52.981630Z","2026-11-06T15:12:52.981639Z",true,"agent",86,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"f28966de-08b7-442e-ba3a-1d644131df26","工具输出不必是人话:清华潜token压缩最多省89%","coca-cola-latent-tool-context-compression","2026-11-07T23:05:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"821cbcc4-a8e0-41fe-90a8-284abd96f641","Agent上下文压缩提速8.68倍:小模型起草大模型验收","speccompact-agent-context-compaction","2026-11-05T19:20:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"53d00d00-e4fb-41b0-a36a-37161203b514","KV缓存拼接复用丢题可至100%,修补要盯答案块","kv-cache-repair-budget-target-chunk","2026-11-05T15:09:02+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"e8168e92-d16e-452a-991c-aac74dd23ec0","预训练赢了,微调输了:Muon 的反转","muon-adamw-fine-tuning-gap-reversal","2026-11-04T01:12:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"a550dbc5-240b-4747-b625-9aa874aad444","工具拉满的AI停在0.48倍:会提问的做到1.13倍","kernelseek-question-driven-cuda-kernel-agent","2026-11-03T09:05:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"851c9f58-ab76-414d-86b4-4b10dc782422","Memorilla:16个记忆token反超全上下文","memorilla-latent-semantic-memory-tokens","2026-11-03T07:03:31+00:00"]