[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-earlyeval-early-stop-agent-eval":3,"news-related-00346b75-f071-42fd-ae16-db4c5569f01a":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"00346b75-f071-42fd-ae16-db4c5569f01a","EarlyEval 提前叫停注定失败的 Agent:近半 token 省下,分辨率只动一两个点","上海交大团队的 EarlyEval 在 Agent 跑分中途用 LightGBM 分类器预判结局并提前叫停:三个基准上砍掉 13%-26% 的步骤、最多省 44.1% 输入 token,预测准确率 89%-97%,对各 agent 解决率的扰动平均只有 1-2 个百分点。","评测 Agent 的账单正在失控。EarlyEval 论文的原话是:让前沿模型在一个 agentic 基准上完整跑一遍,成本可达「数百到数千美元」,而这个价格在迭代开发周期里要反复支付。上海交大团队 9 月 2 日挂上 arXiv 的这项工作给出了一个新解法:不删任务、不换小模型,而是在 agent 跑到一半、结局已经可预判时,提前把它叫停。\n\n## 换一条轴省评测的钱\n\n此前评测降本的主流路线是 benchmark distillation(基准蒸馏)——把成百道题蒸馏成更少的任务,任务数少了,但每个保留任务的执行成本原封不动。EarlyEval 补的是另一条轴:任务内部提前止损。它的核心观察很朴素:agent 的最终结局,往往在执行完成前就从中间行为里显形——一个注定失败的 run,看前几步的行为特征就能判断。\n\n方法上,框架训练了一对 LightGBM 分类器(一个判成功、一个判失败),特征覆盖行为、文本与参考解三族;一旦任一分类器越过校准过的置信度阈值,run 立即中止。论文强调这套监视的单步开销可以忽略。\n\n## 数字:省多少,误差多大\n\n在 SWE-bench Verified、TerminalBench、Toolathlon 三个基准上,论文报告 EarlyEval 可以砍掉 13%-26% 的 agent 步骤,最多省下 44.1% 的输入 token 和 29.4% 的输出 token,同时保持 89%-97% 的结局预测准确率;对每个 agent 的任务解决率(resolve rate)的平均扰动只有 1-2 个百分点。\n\n这个交换比值得细看:对高频内部评测来说,用 1-2 个百分点的榜单噪声换近半的输入 token,在开发迭代场景里几乎一定划算;但如果评测目的是对外发榜、数字要经得起审计,这个扰动就必须谨慎对待。论文自己也把「对解决率的影响」当作核心指标摆出来,而不是藏进附录。\n\n## 开源仓库:代码全给,工件不给\n\n配套 GitHub 仓库(inphotoo\u002Fearlyeval)是一次 code-only release:训练、测试、特征构建、消融、架构对比、策略回放、表格生成代码全部放出,连 TerminalBench 上 4x4 的「换模型 × 换 harness」交叉诊断实验脚本都在;但刻意不含训练好的模型、预测文件、论文表格等生成工件,需要按 README 的复现流水线自建。架构对比部分,仓库在 LightGBM 主模型(I_LightGBM_Dense_AF)之外,还提供了 direct MLP、BERT\u002FCodeBERT、本地 LLM-logit 和 Qwen 微调等基线代码,方便对照复跑。README 的输出路径里出现了 icse_submission_draft 字样,暗示这是投稿 ICSE 的配套代码。\n\n## 所以呢\n\nAgent 评测的经济学正在变成一门独立的学问:训练要钱、推理要钱,现在连「验证训练有没有用」也要按美元计价。EarlyEval 的价值不在于那个 44.1%,而在于它把「评测预算」从常量变成了可调参数——想省钱就提前停,要精确数字就等跑完。对每天在 SWE-bench 上烧钱的团队来说,这可能是最务实的一类研究。论文上线当天在 Hugging Face Daily Papers 收获 110 个推荐(单一社区榜单口径)。\n\n参考:arXiv:2609.02783 — https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.02783","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.02783","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"9abc7712-fd31-4d91-88f0-4365da383c38","en","EarlyEval Halts Doomed Agent Runs to Cut Eval Token Cost 44%","EarlyEval stops agent runs early once the outcome is predictable: up to 44.1% input tokens saved at 89-97% accuracy, resolve rates shift 1-2 points.","The bill for evaluating agents is spiraling. The EarlyEval paper puts it plainly: a single pass of a frontier model over an agentic benchmark can cost hundreds to thousands of dollars — a price paid repeatedly across iterative development cycles. The Shanghai Jiao Tong University team, whose report landed on arXiv on Sep 2, offers a different fix: don't drop tasks, don't switch to a smaller model — just halt the agent run the moment its outcome becomes predictable.\n\n## A new axis for cheaper evaluation\n\nThe mainstream route to cheaper evaluation so far has been benchmark distillation: compress hundreds of tasks into fewer, which shrinks the task count but leaves the cost of each retained task untouched. EarlyEval attacks a complementary axis — early stopping inside each task. Its core observation is simple: an agent's final outcome is often evident from its intermediate behavior well before execution completes; a doomed run shows it in the first few steps.\n\nMethod-wise, the framework trains a pair of LightGBM classifiers (one for success, one for failure) over behavioral, textual, and reference-solution features. The moment either classifier crosses a calibrated confidence threshold, the run is halted. The per-step overhead of this watch, the paper notes, is negligible.\n\n## The numbers: how much saved, at what distortion\n\nAcross three benchmarks — SWE-bench Verified, TerminalBench, and Toolathlon — the paper reports EarlyEval eliminates 13%-26% of agent steps and up to 44.1% of input tokens and 29.4% of output tokens, at 89%-97% prediction accuracy, while perturbing per-agent resolve rates by only one to two percentage points on average.\n\nThat trade deserves a closer look: for high-frequency internal evaluation, swapping 1-2 points of leaderboard noise for nearly half the input tokens is almost certainly a win; for public leaderboard submissions whose numbers must survive audit, the perturbation has to be handled with care. Notably, the paper reports resolve-rate impact as a headline metric rather than burying it in an appendix.\n\n## The repo: all code, no artifacts\n\nThe companion GitHub repository (inphotoo\u002Fearlyeval) is a code-only release: training, testing, feature construction, ablation, architecture comparison, policy replay, and table-generation code are all there, including the 4x4 model-by-harness cross-diagnosis scripts for TerminalBench. Trained models, prediction files, and paper tables are deliberately excluded — you rebuild them via the README's reproduction pipeline. For architecture comparison, the repo ships baseline code for direct MLP, BERT\u002FCodeBERT, local LLM-logit, and Qwen fine-tuning alongside the main LightGBM model (I_LightGBM_Dense_AF). The output paths in the README contain an icse_submission_draft segment, hinting this is the companion code for an ICSE submission.\n\n## So what\n\nThe economics of agent evaluation is becoming its own discipline: training costs money, inference costs money, and now verifying whether any of it worked is priced in dollars too. EarlyEval's real contribution is not the 44.1% figure — it is turning evaluation budget from a constant into a dial: stop early when you are iterating, run to completion when you need the exact number. For teams burning cash on SWE-bench daily, this may be the most practical kind of research. The paper gathered 110 upvotes on Hugging Face Daily Papers on its first day (a single community board).\n\nReference: arXiv:2609.02783 — https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.02783","earlyeval-early-stop-agent-eval","2026-09-03T21:04:52Z","2026-09-03T21:05:01.713916Z","2026-09-03T21:05:01.713925Z",true,"agent",56,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"10bd728a-5d05-461b-ba14-7c6ea51f8ce3","HyperTool 把 MCP 工具调用「折叠」成一段代码，Qwen3 智能体准确率直接翻倍","hypertool-mcp-folded-code-qwen3-2x","2026-06-14T20:00:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"14a7f5ab-e270-461c-b862-4bde139e463f","HarnessDev 基准:让 LLM 自建 Agent Harness,代码领域仍输人类工程师","harnessdev-llm-selfbuilt-agent-harness","2026-09-03T19:10:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"089195fb-7fe5-4ba9-a4bc-8e356fe5e923","BAAI把1000个GitHub仓库蒸馏成5000个技能,科研agent奖牌率31%冲到73%","baai-disco-repo-to-skill-library","2026-09-03T17:07:35+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"ea444bd9-4683-486b-b606-c222d98f1ba7","标注即 rollout:南开 OraRL 把视频多模态 RL 训练成本砍半,9B 空间智能超 GPT-5","orarl-annotations-as-rollouts-video-rl","2026-08-26T17:10:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"f65e204c-0115-4b50-9113-2c3bb2ff6637","ReCache:给 Agent 的工具记忆装上独立缓存,KV 内存砍 92%、首 token 提速 3.655 倍","recache-agent-kv-cache-reuse","2026-08-24T15:30:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"c4ec4625-4a84-4c24-88f0-0ef1beb4f19e","Grok 4.6 发布:61 分追平 GPT-5.6 Sol,把长程 Agent 的 token 账单砍到四分之一","grok-4-6-agentic-cost-frontier","2026-08-14T19:00:00+00:00"]