[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-llm-judge-batch-grading-instability":3,"topics-all":38,"news-related-4c0315b0-aae1-4483-ab12-3281f8b729d7":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"4c0315b0-aae1-4483-ab12-3281f8b729d7","批量打分翻车:LLM 判决 11%–22% 会翻转","批量评审是 LLM 流水线的省钱标配:25 条样本塞进一个 prompt,让模型一次返回 25 个 JSON 判决。慕尼黑工业大学的一篇论文(arXiv:2610.40241)发现,同样这 2000 条样本换个随机分组,11%-22% 的判决就会翻转,而 AUROC 只差 0.033。","把 25 条样本塞进一个 prompt、让模型一次返回 25 个 JSON 判决，这是 LLM 评审流水线的省钱标配。它默认了一个没人明说的前提：某条样本的判决只取决于它自己，不受同批邻居和它在 prompt 里位置的影响。慕尼黑工业大学的一篇论文测了测这个前提，结论不太乐观。\n\n论文标题很直白，叫「被同批的样本带着判」（[arXiv:2610.40241](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.40241)）。它在社交媒体有害性检测、学生简答评分、评论星级三个任务上，用五个 4B 到 9B 的开源权重模型各评了 2000 条样本，每批 25 条。结果是：在有害性和简答评分两个任务上，换个随机分组重新分批，同一批样本的判决有 11%–22% 会翻转，而 AUROC 几乎不动。\n\n## 省下来的钱，是判断力的代价\n\n每条请求都要重复同一段指令，把 25 条样本拼进一个 prompt，token 成本能省下 80% 以上。论文引用的既有研究（Pipal 等，2026）称，25 到 100 条一批、精度损失低于 2 分；论文引用的厂商文档也建议在非实时的内容审核场景这样用。实践者照做不奇怪。\n\n问题在于这个前提从来没人写下来：判决必须是「可分离的」（separable）——任意一条样本在任意上下文里，得到的等级都得一样。decoder-only 模型没有任何机制保证这点。模型写第 12 条样本的判决时，前 24 条文本和它自己刚写下的 11 个判决，全都躺在上下文里。\n\n## 对照组是怎么设的\n\n三个任务各有 2000 条样本。有害性用 Measuring Hate Speech 语料，五级量表，判到「侮辱或针对个人的骚扰」及以上算命中；简答评分用 SciEntsBank，判成错、部分对、对；星级用 Yelp 评论，判到四星及以上。五个模型是 Qwen3-4B-Instruct-2507、Qwen3-8B（关闭思考）、Gemma-4-E4B-it、Granite-3.3-8B-Instruct、GLM-4-9B-0414，bf16 精度、贪心解码，每个模型配一张 RTX 5090。\n\n关键在对照组的设计。作者不只跑「换一批」，还跑了三条基线：改样本键名、改写指令措辞、以及把同样的 prompt 改成一条一条生成，用来量纯数值噪声。这些基线量出 4.2%–8.4% 的判决变化，其中纯数值噪声只占 0.5%–2.1%。而换随机分组，变化是 11%–22%，比基线里最高的那条还高出 6.5 到 13.3 个百分点。\n\n## 准确率为什么骗人\n\n最值得警惕的是：批量评审的 AUROC 相比逐条评分只变化 -0.033 到 +0.010，任何验证阈值都拦不住。更要命的是工作点会漂：在有害性任务上，Gemma-4 逐条评时会标出 49% 的评论，批量评只标 31%；Granite 从 59% 掉到 43%。同一批评论，标记比例差出十几个点。\n\n论文的解释是：在阈值模型下，上下文贡献的方差让分数的 AUROC 按比例下降，却让决策的翻转率按方根增长。「对分数影响小」和「对决策影响大」可以同时成立，因为翻转率取决于阈值附近的分数密度。\n\n模型更大也没救。附录里四个 14B 到 32B 模型换批后仍有 6%–15% 的判决变化，比基线高 4.2–9.1 个点。\n\n## 拆开看：顺序、位置、同批\n\n论文把翻转率精确分解成三个成分：邻居的排列顺序占潜在差异的 0.42–0.69，样本所在位置占 0.09–0.32，「和谁同批」这个成员效应占 0.01–0.25（均值 0.15）。在决策尺度上，顺序贡献 0.62–0.80 的翻转，成员效应只占 0.00–0.14。\n\n那为什么「换一批」这么要命？顺序和位置至少能被投票平均掉，「谁跟谁同批」却不行。作者做了个更狠的实验：每个批次按九种顺序各评一次再取多数票，仍有 52%–60% 的翻转残留；换成十五个独立随机批次再取多数票，跟逐条评分相比仍有 8.4%–17.8% 的判决不一致。\n\n机制上，论文在生成中途动刀做了两次干预：把别的批次的判决换进来，7%–23% 的判决跟着变；把别的批次的文本换进来，9%–26% 跟着变。两条通道同时起作用——模型既被邻居的文本带跑，也被自己刚写下的判决带跑。\n\n星级任务是个反例：模型与真值 AUROC 0.96，写出的星级虽有 16%–21% 变化，决策却只变 2.9%–3.9%。任务越模糊，影响越大。\n\n## 那怎么办\n\n论文的建议很直接：模棱两可的样本一条一个 prompt 评，靠缓存共享前缀把成本补回来；用了批量，就顺手报告一下与逐条评分的一致率。\n\n更该被记住的是「多数投票不救场」。BatchPrompt 这类做法一直用排列投票压位置效应，看着像通用解，论文测出来它对「同批成员」几乎无效。而这恰恰最容易踩的一层——谁和谁被放进同一个 prompt，通常由业务逻辑顺手决定，不是评测设计安排的。\n\n整个 LLM-as-a-judge 的结论有多少经得起重跑？另一项独立研究（[arXiv:2610.31172](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.31172)，跨 27 种语言、9 种提示策略、4 个开源评审模型）给出的答案是：提示策略不可迁移，选评审模型比选提示策略更重要，同一策略在不同评审模型上效果甚至反向，而且这些不稳定在低资源语言里更严重，还被汇总口径掩盖。\n\n两篇论文测的是不同层面，却指向同一件事：聚合数字下面压着两层噪声——评审模型本身的选择，以及评审时上下文怎么组。现在很多「评测分数」，更像是某次具体采样过程的产物，而不是被评估对象的能力。\n\n所以下次看到一份榜单，除了看分数，也值得看一眼：这些判定是一条一条评出来的，还是拼成批一起评的？如果是后者，那分数里有多少来自被评估对象、有多少来自批次的组合方式，报告本身恐怕说不清。\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.40241","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"5e628969-6d2a-437f-998a-104e4b16cfb1","ai-progress",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"199238e9-ab38-45f4-b827-8786e6c29786","en","Batch Judging: Regrouping Flips Up to 22% of LLM Verdicts","Batching 25 items saves tokens but couples verdicts: regrouping 2,000 items flips 11-22% of decisions while AUROC barely moves, a TUM study finds.","Packing 25 items into one prompt and asking the model to return 25 JSON verdicts in a single pass is the standard cost-saving trick in LLM review pipelines. It rests on an assumption nobody writes down explicitly: that a verdict depends only on the item itself, not on the neighbours sharing the prompt or on the item's position inside it. A paper from the Technical University of Munich put that assumption to the test, and the result is not flattering.\n\nThe title says it plainly: judged in company ([arXiv:2610.40241](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.40241)). Across three tasks - harm detection in social-media comments, short-answer grading, and review star ratings - the authors ran five open-weight models of 4B to 9B parameters over 2,000 items each, batching 25 items per prompt. The finding: reshuffling the same 2,000 items into different random batches flips 11% to 22% of the decisions, while AUROC barely moves.\n\n## The money saved costs you judgement\n\nFirst, why this practice spread. Every request repeats the same instruction block, so stitching 25 items into one prompt cuts token cost by more than 80%. A prior study cited in the paper (Pipal et al., 2026) reports batches of 25 to 100 items with up to ten coded variables per prompt, saving more than 80% of tokens while losing less than 2 points of accuracy. Vendor documentation referenced by the paper recommends batching for moderation that need not happen in real time. When both vendors and academia say so, practitioners following it is unsurprising.\n\nThe problem is that the condition this practice depends on was never written down: verdicts must be separable. The paper's definition is blunt - for any item, in any context, the level assigned must be the same. No decoder-only model enforces that. When the model writes the verdict for item 12, the other 24 texts and the 11 verdicts it has already written are all sitting in its context.\n\n## How the controls were built\n\nEach task contributes 2,000 items. Harm uses the Measuring Hate Speech corpus with a five-level scale, counting a hit at level 2 (insults or harassment aimed at a person) and above; grading uses SciEntsBank, labelled incorrect, partial, or correct; stars uses Yelp reviews, counted as a hit at four stars or above. The five models are Qwen3-4B-Instruct-2507, Qwen3-8B with thinking disabled, Gemma-4-E4B-it, Granite-3.3-8B-Instruct, and GLM-4-9B-0414, run in bf16 with greedy decoding, one RTX 5090 per model.\n\nThe controls are the interesting part. Beyond reshuffling, the authors ran three baselines: renaming the item keys, rewording the instruction, and generating the same prompts one at a time to isolate pure numerical noise. Those baselines move 4.2% to 8.4% of decisions, and numerical noise alone accounts for only 0.5% to 2.1%. A new random partition moves 11% to 22%, which is 6.5 to 13.3 points above the largest baseline.\n\nRenaming keys and rewording instructions are surface perturbations. Changing which items share a prompt is a different thing entirely.\n\n## Why accuracy hides it\n\nThe most worrying part: AUROC for batched grading differs from single-item grading by only -0.033 to +0.010, a shift too small for any conventional validation threshold to catch. Worse, the operating point drifts. On the harm task, Gemma-4 flags 49% of comments when grading one at a time but only 31% when batching; Granite drops from 59% to 43%. The same comments, flagged at rates that differ by more than ten points.\n\nThe paper supplies the mechanism. Under a threshold model, variance contributed by context lowers the AUROC of scores in proportion to its share, but flips decisions in proportion to its square root. \"Small effect on scores\" and \"large effect on decisions\" can hold at once, because the flip rate depends on how densely the scores cluster near the threshold.\n\nScaling up does not rescue it. In the appendix, four models from 14B to 32B parameters still show 6% to 15% of decisions changing across batches, 4.2 to 9.1 points above baseline.\n\n## Taking it apart: order, slot, company\n\nThe paper decomposes flip rates exactly into three components. The order of the neighbours accounts for 0.42 to 0.69 of the latent dissimilarity, the item's slot for 0.09 to 0.32, and membership - which items share the prompt - for 0.01 to 0.25 (mean 0.15). On the decision scale, order accounts for 0.62 to 0.80 of the flips while the membership main effect accounts for only 0.00 to 0.14.\n\nSo why does reshuffling bite so hard? Because order and slot can at least be averaged away by voting, while who shares the batch cannot. The authors ran a harsher experiment: grade every batch in nine different orders and take the majority vote, and 52% to 60% of the flips survive. Replace the batch with fifteen independent random batches and take the majority of those, and 8.4% to 17.8% of decisions still disagree with single-item grading.\n\nMechanically, the paper intervenes mid-generation: swapping in another batch's verdicts changes 7% to 23% of decisions, and swapping in another batch's texts changes 9% to 26%. Two channels act at once - the model is pulled around both by neighbouring texts and by the verdicts it has already written itself.\n\nThe stars task is a counterexample. With AUROC 0.96 against ground truth, the written rating changes for 16% to 21% of reviews, yet only 2.9% to 3.9% of decisions change. The more ambiguous the task, the larger the effect.\n\n## What to do instead\n\nThe recommendation is direct: grade ambiguous items one per prompt and recover most of the cost with a cached prefix, and whenever items are batched, report agreement with single-item grading.\n\nWhat is worth remembering is really the finding that majority voting does not save you. Approaches like BatchPrompt have long used voting over permutations to damp position effects, which looks like a general fix, but the paper measures it as nearly ineffective against the membership layer. That is precisely the layer practitioners trip over most easily: who gets grouped with whom in one prompt is usually decided by whatever business logic is convenient, not by deliberate evaluation design.\n\nThis points at a larger question. How many LLM-as-a-judge conclusions survive a re-run? An independent study ([arXiv:2610.31172](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.31172), spanning 27 languages, 9 prompting strategies, 4 open-weight judges, 13 datasets, and over a million pointwise scores) finds that prompting strategies are not transferable, that the choice of judge model dominates the choice of prompting strategy, that a strategy's effect can even reverse across judges, and that this instability worsens in low-resource languages while aggregate reporting conceals it.\n\nThe two papers probe different layers, yet land in the same place: underneath the aggregate number sit two layers of noise - the choice of judge model, and how the context was assembled during judging. Many \"evaluation scores\" today are better described as artefacts of one particular sampling run than as a property of the system being evaluated.\n\nSo the next time you read a leaderboard or a review report, look past the score and check one thing: were these verdicts produced one at a time, or in a batch? If the latter, how much of the score came from the system under test and how much from the composition of the batch is something the report itself probably cannot tell you.","llm-judge-batch-grading-instability","2026-10-30T07:28:14Z","2026-10-30T07:29:30.044022Z","2026-10-30T07:29:30.044031Z",true,"agent",82,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"1b51020b-9a0c-46f8-98ec-e4e0c8c051a6","基准分其实是难度的加权平均:25 个主流基准的体检报告","benchmark-difficulty-measure-coverage-gaps","2026-11-01T01:16:42+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"c780f08d-fa7e-4c4b-8cf5-ee367bde4c06","零 RL 涨 42.9 分几乎不说「等等」:它来自训练数据","reasoning-wait-marker-provenance-rl-training-data","2026-10-25T05:15:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"c368ad9f-9308-4a0b-8f5c-3ae4601b48b9","D2K-Bench: 专家设计把 LLM 写 GPU 核提速 33.9%","d2k-bench-llm-gpu-kernel-design-guidance","2026-10-07T03:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"43eda321-b0b7-4df7-b20e-9758cbab42c9","记忆越完整,眼前题越做不对:MemTrapBench 把 LLM 长期记忆框架打回原形","memtrapbench-llm-memory-cognitive-traps","2026-08-22T04:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"88105269-9641-44c3-a705-1cf07314614f","LLM 思维链能看出\"用了几分力\":SARE 给每一步推理做 CT 扫描","step-aware-reasoning-energy-llm-cot","2026-08-04T04:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"95b04c15-d5e5-4dca-ab2b-14e343bdd4e6","UC Berkeley 曝光 AI 基准测试系统性漏洞：45 种方法可在 13 个主流榜单上「不解决任何问题拿满分」","uc-berkeley-benchmark-45-cheats-13-leaderboards","2026-05-15T01:00:00+00:00"]