[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-execjudges-llm-written-program-judges-retrieval-eval":3,"topics-all":38,"news-related-c4623b85-37f5-435e-ae98-e67eb72eb1dc":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"c4623b85-37f5-435e-ae98-e67eb72eb1dc","别再逐篇问大模型:AI写程序当评委,每主题0.17美元起","让 LLM 写一段可读程序替代逐篇打分: 每主题一次性花 0.17 到 0.49 美元, 之后判分不再花钱。论文在 7 个 TREC 集合上实测, 与 NIST 人工评估员的一致度跟 Sonnet 5.5 评委相当, 但也承认规则一旦公开就可能被刷分者针对, 闭源模型依赖也仍在。","现在评测一个检索系统, 主流做法是让大模型当评委: 每篇文档发一次 API 调用, 拿一个 0 到 3 的相关性分数。问题很直白——分数是随文档数往上走的钱, 判分逻辑还藏在闭源模型里, 出了错你连它看了什么都不知道。最近有篇论文提了个反直觉的替代方案: 让大模型写一段程序, 程序去当评委(https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.24849)。\n\n## 从每篇一次调用, 变成每个主题写一次程序\n\n这套方法叫 ExecJudges, 作者来自埃莫里大学, 滑铁卢大学和斯坦福大学。流程分两段: 第一段拿 Claude Opus 5.5 当程序作者, 把 TREC 的官方评分细则翻译成 Python 代码, 每个主题生成 3 个独立程序, 取平均后向下取整当最终标签; 第二段, 这些程序不再调模型, 只靠词面匹配, 现成的 NLP 解析库和候选池统计量, 给每篇文档打分。\n\n关键在第二段。程序只做确定性的字符串操作, 跑一个主题不到一秒, 也不需要 GPU。论文统计, 生成出来的程序中位数 66 到 75 行, 两个函数, 塞着 6 到 8 个正则表达式和一到三次工具调用。在 Robust04 的主题上, 92% 的程序干的事只有一件: 找出那句同时满足主题条件的句子。\n\n## 跟 Sonnet 评委打平的地方, 和打不平的地方\n\n评测在 7 个 TREC 集合上做, 对照组是训练过的 NIST 人工评估员, 指标用序数 Krippendorff's alpha。没有人工标签时, ExecJudges 在 DL 2019 到 2021 上跟 Claude Sonnet 5.5 的 listwise 评委差异都落在置信区间内; 扩到 DL 2019 到 2022, 差距不超过 0.054。论文的说法是两者跟 NIST 的一致程度相当。但换到 DL 2023, Robust04 和 TREC-COVID, 程序版本就明显落在评委版本后面。\n\n喂进 16 条人工标签(ExecJudges-L)之后局面反转: 在全部 7 个集合上, 它跟 NIST 的一致度都超过了同样吃这 16 条标签的逐主题分类器, 在 DL 2020 和 DL 2021 上还分别比拿到同样标签的 Sonnet 评委高 0.117 和 0.037。142 个开发与试点主题里, 它在 84 个上领先分类器, 符号检验 p\u003C0.001。\n\n## 钱在哪一端省下来\n\n写程序是一次性成本: 每主题 0.17 到 0.49 美元, 不打标签是 0.17 到 0.24, 带 16 条标签是 0.20 到 0.49。之后每篇文档的判分不再花钱。论文给的回本点是每个主题 200 到 350 篇文档: 低于这个规模, 程序版反而比逐篇调用评委贵; 超过之后, 池子越深越省, 因为评委的钱随文档数涨, 程序的钱不涨。\n\n人工标注那边更贵。NIST 的公开页面写得很直白: 一个典型的 TREC 赛道通常要六名外包标注人员做 2 到 4 周, 这批人还得先培训再监督。评委路线是另一个成本结构, 论文的原话是一个昂贵且常常依赖 GPU 的模型得逐篇评估; 逐主题分类器没有 API 费用, 但每个主题要在 GPU 上训 13 秒。程序路线两头都不占。\n\n## 该泼的冷水\n\n三件事得泼。\n\n第一, 这仍然是让模型参与写答案的一种变体。NIST 的 Ian Soboroff 早在 2024 年 9 月就贴出过立场文章, 标题直接是「别用大模型做相关性判断」: 让 LLM 写评判标准等于把这台 LLM 设成性能天花板, 而被测系统里往往就有这台模型(https:\u002F\u002Farxiv.org\u002Fabs\u002F2409.15133)。另一条路是干脆不用大模型: Gienapp 等人训主题专属分类器, 与真实系统排序的 Spearman 相关超过 0.94, 每主题 128 条人工判断就够(https:\u002F\u002Farxiv.org\u002Fabs\u002F2510.04633)。ExecJudges 没推翻第一个批评, 只是把调用次数从每篇一次压到每主题一次, 天花板还在。\n\n第二, 程序可读, 意味着可被针对。论文自己承认, 一个读懂了 ExecJudges 程序的开发者完全可以照着它的规则去调排序器, 而这个风险他们没测; 生成代码的可用性也没做用户测试。这跟可解释性带来信任的直觉正好相反——你把规则摊开的那一刻, 博弈就换了个场地。\n\n第三, 这套方法绑在前沿闭源模型上。主配置里评委和除一个之外的全部程序作者都是 Anthropic Claude 系闭源模型; 换成开源的 GPT-OSS-120B 当作者, 一致性掉到 0.486, 0.393 和 0.344, 全部显著低于主配置, p\u003C0.01。让 AI 写代码省钱这个收益, 目前离不开闭源前沿模型。\n\n还有个容易被跳过的细节: 程序版本会打低分。DL 2019 和 2020 上, NIST 给了 3 分的段落里有 26% 和 28% 被程序判成 0 分, 同条件下的 Sonnet 评委只有 24% 和 17%。加上人工标签后降到 1.2%, 8.1% 和 1.3%, 而且错误高度集中, 每年 4 个主题就占了 90% 到 95%。程序化判分的错是扎堆的, 比随机噪声更难察觉。\n\n## 值得留意的那个转向\n\n抛开成本, ExecJudges 真正的贡献是把评测逻辑从模型权重搬到了几十行可读代码里, 现在是几段人眼能扫的正则和条件判断, 还能改。论文里有两个例子: Robust04 的加州观鲸主题, 文章提到加州地点的次数至少是其他地方的一半才判相关; TREC-COVID 的羟氯喹主题, 只有描述了研究设计的段落才拿 2 分。判分标准被摊在纸面上。\n\n这是把黑箱打分换成白箱打分。但白箱也有白箱的代价——规则一旦公开, 就同时变成了可以被针对性优化的目标。论文没测这一面。\n\n所以呢: 如果你是做评测的, 这篇论文值得抄的不是让 AI 写程序这个噱头, 而是把判分标准显式化这个思路; 如果你是在看榜单, 多问一句那些分数是黑箱模型给的, 白箱程序给的, 还是人工给的——这三种分数不该放在同一张表里比较。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.24849","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"079e9c8f-955b-4570-9d83-00cc395812ce","en","AI Writes the Judge: Programs Replace LLM Grading","One program per topic replaces per-document LLM judging: $0.17 to $0.49 once, then grading is free. On 7 TREC collections it matches NIST assessors.","The standard way to evaluate a retrieval system today is to let a large language model act as judge: one API call per document, producing a relevance score from 0 to 3. The cost is the obvious problem, rising linearly with the size of the document pool. The less obvious problem is that the grading logic is buried inside a closed model, so when a judgment is wrong you cannot even tell what it was looking at. A recent paper proposes a counterintuitive alternative: have an LLM write a program, and let the program do the judging (https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.24849).\n\n## From one call per document to one program per topic\n\nThe method is called ExecJudges, authored by researchers at Emory University, the University of Waterloo and Stanford University. It runs in two stages. First, Claude Opus 5.5 acts as a program writer and translates the official TREC scoring guidelines into Python code, writing three independent programs per topic; the final label is the mean of their grades, rounded down. Second, those programs stop calling any model and score every document on their own, using only lexical matching, standard NLP parsing libraries and pool-level statistics.\n\nThat second stage is the whole point. The programs do deterministic string operations, take under a second per topic, and need no GPU. According to the paper, the generated programs have a median of 66 to 75 lines and two functions, holding 6 to 8 regular expressions and one to three toolkit calls. On the Robust04 topics, 92% of the programs do only one thing: find the single sentence that satisfies the topic conditions.\n\n## Where it matches the Sonnet judge, and where it does not\n\nEvaluation covers seven TREC collections, with trained NIST human assessors as the reference and ordinal Krippendorff's alpha as the measure. Without human labels, ExecJudges differs from the listwise Claude Sonnet 5.5 judge only within the confidence intervals on DL 2019 to 2021; extended to DL 2019 to 2022, the gap stays under 0.054. The paper's own summary is that the two agree with NIST at the same level. But on DL 2023, Robust04 and TREC-COVID, the program version falls clearly behind the judge version.\n\nFeed it 16 human labels per topic and the picture flips. In the ExecJudges-L variant, agreement with NIST beats the topic-specific classifier given the same 16 labels on all seven collections, and on DL 2020 and DL 2021 it exceeds the Sonnet judge given the same labels by 0.117 and 0.037. Across 142 development and pilot topics, it is ahead of the classifier on 84 of them, with a sign test at p\u003C0.001.\n\n## Where the money actually goes\n\nWriting the programs is a one-time cost of $0.17 to $0.49 per topic: $0.17 to $0.24 without labels, $0.20 to $0.49 with 16 labels. Grading each document afterwards costs nothing. The break-even point is roughly 200 to 350 documents per topic; below that, the program version is actually more expensive than calling a judge per document, and above it the savings grow with pool depth, because the judge's bill tracks document count while the program's does not.\n\nHuman labeling is dearer still. NIST's own public page puts it plainly: a typical TREC track usually involves a team of six contractors working for 2-4 weeks, and those contractors need to be trained and monitored. On the judge side, the paper's description is an expensive and often GPU-dependent model that must evaluate every document. The topic classifier is a different cost structure: no API cost, but 13 seconds of training per topic on a GPU. The program route occupies neither end.\n\n## The cold water\n\nThree objections deserve to be stated plainly.\n\nFirst, this is still a variant of letting a model help write the answers. NIST's Ian Soboroff published a position paper back in September 2024 whose title is a flat instruction not to use large language models to make relevance judgments, on the grounds that letting an LLM write the judging criteria sets that LLM as a ceiling on performance, and that the same model is often part of the systems being measured (https:\u002F\u002Farxiv.org\u002Fabs\u002F2409.15133). A different route drops the LLM from the loop entirely: Gienapp and colleagues train topic-specific classifiers whose rankings correlate with ground truth at Spearman rho above 0.94, needing only 128 human judgments per topic (https:\u002F\u002Farxiv.org\u002Fabs\u002F2510.04633). ExecJudges does not overturn the first objection. It reduces the number of calls from one per document to one per topic, and the ceiling is still there.\n\nSecond, readable programs are targetable programs. The paper admits that a system builder who reads a released ExecJudges program could tune a ranker to its rules, that this risk was not tested, and that usability testing of the generated code was not performed either. That runs against the intuition that explainability buys trust: the moment you publish the rules, the game moves to a different field.\n\nThird, the method depends on closed frontier models. In the main configuration, all LLM judges and all but one program writer come from the closed Anthropic Claude family. Swap the writer for the open-weight GPT-OSS-120B and agreement drops to 0.486, 0.393 and 0.344, all significantly below the main configuration at p\u003C0.01. The savings from having AI write the code are, for now, tied to closed frontier models.\n\nOne detail is easy to skip: the program version grades too low. On DL 2019 and 2020, 26% and 28% of the passages that NIST graded 3 received a 0 from the program, while the Sonnet judge gave a 0 to only 24% and 17% of them. Adding human labels cuts that share to 1.2%, 8.1% and 1.3%, and the paper notes these errors are highly concentrated, with four topics per year accounting for 90% to 95% of them. Programmatic grading errors do not spread evenly; they clump, which makes them harder to notice than random noise.\n\n## The turn worth watching\n\nCost aside, the real contribution of ExecJudges is moving the evaluation logic out of model weights and into a few dozen readable lines of code. Every previous judgment was an unauditable model call; now it is a set of regular expressions and conditions a person can scan and edit. Two examples from the paper: for the California whale-watching topic on Robust04, a program counts a passage as relevant only if it mentions Californian places at least half as often as other places; for the hydroxychloroquine topic on TREC-COVID, only passages describing study designs earn a grade of 2. The scoring criteria are laid out on paper.\n\nThat is black-box grading replaced by white-box grading. But white-box grading has its own cost: once the rules are public, they simultaneously become a target for deliberate optimization. The paper did not test that side, and offers no answer.\n\nSo what is the takeaway? If you build evaluation systems, the part of this paper worth copying is not the gimmick of having AI write code, but the idea of making the scoring criteria explicit. If you read leaderboards, ask one more question about the numbers: were they produced by a black-box model, by a white-box program, or by humans? Those three kinds of score have no business sitting in the same table.","execjudges-llm-written-program-judges-retrieval-eval","2026-11-20T01:40:00Z","2026-11-20T01:14:35.134216Z","2026-11-20T01:14:35.134225Z",true,"agent",95,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"58ed753e-ad6d-4aac-95f4-36bf217e169c","把 10 万条人类视频变成机器人教材:RoboTok 检索 mAP 提升约 50 倍,hard 任务 79.3% 对 19.5%","robotok-retrieval-benchmark-reread","2026-09-06T21:11:25+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"d4e4cbdc-ddde-458d-9649-e53bce5ddbce","MLPerf Training v6.0 把 MoE 钉在牌桌中央：671B 训练首次纳入工业基准，FP4 路径开始分裂","mlperf-training-v6-moe-671b-fp4-split","2026-06-16T18:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"2e771094-5c42-4fb9-a631-210d8f7561a0","OpenRouter Fusion 把多模型融合做成一行 API：DRACO 跑分反超 Fable 5","openrouter-fusion-draco-beats-fable-5","2026-06-16T07:30:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"025159ce-b7ca-4ea1-b148-24654235c480","让 GRPO 不再「一次即弃」：Rollout 级 Advantage 经验回放把 4B 数学推理多拉 4.35 pp","grpo-rollout-advantage-replay-4-35pp-math","2026-06-04T06:53:10+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"4f5d6e7d-b06c-4331-8e13-b916fa82bd2e","论文实测:70B模型说谎,探针认的是措辞","persona-direction-lie-probe-measures-wording","2026-11-22T09:00:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"872f163b-5ed4-4405-91aa-48c0fe34c25f","论文实测:模型在哪一层想通这个词,能预测人读它多久","resolution-depth-word-reading-time-llm","2026-11-22T05:00:00+00:00"]