[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-inferencebench-open-ended-llm-optimization":3,"news-related-7d371b09-9792-465d-b73a-3d0af4735129":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"7d371b09-9792-465d-b73a-3d0af4735129","InferenceBench：15 个前沿 Agent 自主做 LLM 推理优化","ICML 2026 收录的 InferenceBench(arXiv:2607.20468)把「部署一个 OpenAI 兼容推理服务器并压榨延迟\u002F吞吐」丢给前沿编程 Agent 自主完成。15 个 Agent 配置在 1 张 H100、2 小时预算下,4 个场景跑下来:相对原生 PyTorch 基线平均加速 8.08 倍,优于 vLLM 等推理引擎默认设置 4.05 倍,但仍比同预算下的简单超参搜索慢 11.53 倍;85.3% 的提交落在 vLLM 上,Agent 嘴上会列 chunked prefill\u002F量化\u002F投机解码,但搜索浅、最终提交常丢弃已找到的最优配置。该基准由 ELLIS Institute Tübingen 的 Jehyeok Yeon 等团队发布,代码与 leaderboard 已开源。","# InferenceBench:把 LLM 推理优化交给 AI Agent 自己做,跑出来一份并不漂亮的成绩单\n\n> 当你让一个「会写代码的 AI Agent」独自面对 1 张 H100、2 小时、一台要部署到线的 OpenAI 兼容推理服务器,它到底能优化到什么程度?ICML 2026 上线的 InferenceBench 给了一份相当冷静的答案。\n\n## 一句话先讲清这篇在讲什么\n\nInferenceBench 是来自 **ELLIS Institute Tübingen \u002F Max Planck Institute for Intelligent Systems \u002F Tübingen AI Center** 的 Jehyeok Yeon、Ben Rank、Maksym Andriushchenko 提出的一个**开放式基准**,收录于 **ICML 2026**(arXiv:2607.20468,2026-05-20 提交),官方代码与 leaderboard 开源在 。它的核心动作是:**把一个目标 LLM、一张 H100、一个 2 小时时钟预算,以及四个互相隔离的优化场景丢给前沿编程 Agent,让它从零搭出一个 OpenAI 兼容的推理服务器,然后只对它「最终提交的那一份」打分**。提交的服务器还要过正确性检查和 reward-hacking 完整性审计,任何失败、不可达或回退到 PyTorch 基线以下的最终提交,统一按 PyTorch 基线计分,中间过程的成绩不算。\n\n## 这四个场景是这么切的\n\n为了让 agent 的优化目标彼此可分,InferenceBench 拆出了四条独立的「瓶颈赛道」:\n\n1. **Prefill Latency**:长上下文 prompt,衡量 **time-to-first-token(TTFT)**。\n2. **Decode Latency**:长生成,衡量每个 output token 的时间。\n3. **Throughput**:并发流量,涵盖 **burst \u002F Poisson \u002F constant-rate** 三种请求剖面。\n4. **All-In-One**:综合场景,延迟与吞吐指标做几何均值。\n\n每个场景下,Agent 需要在不知道「最优配置」的前提下,自己装引擎、调 flag、跑基准、决定保留还是回滚。基准明确告诉 agent:不能引入第三方预量化检查点,也不能修改评估 harness —— 这两点是为了防止「借力记忆」而非「真优化」。\n\n## 15 个前沿 Agent 跑下来的成绩单\n\n论文与官方 leaderboard(截至 2026-08 的 v1.0.4 版本,刚把 Cost 列与 Grok 4.5\u002F4.6 加进来)给出的几个核心数字:\n\n- **相对原生 PyTorch 基线**,Agent 普遍能拿到 **最高 8.08 倍**的加速 —— 比什么都不调的 baseline 强得多。\n- **相对 vLLM、SGLang、TGI 等推理引擎的默认配置**,Agent 也能 **平均高约 4.05 倍**。\n- **但相对同时间预算下的简单超参搜索(直接调 vLLM\u002FSGLang\u002FTGI 的运行参数与 CLI flag)**,Agent 反而 **落后最多 11.53 倍**。\n- 跨 15 个前沿 Agent 配置的整体几何均值,在 forced-engine 对比里:**TGI-only 8.31×、SGLang-only 7.69×、vLLM-only 6.17×**,而**非 agent 的 per-scenario best search 跑到 14.30×**,依然没人能追上。\n\n换句话说:**Agent 是「高手」,但不是「最强工程师」**。给它 2 小时,它能干翻裸 PyTorch,也能干翻推理引擎默认设置,但只要让一个会调超参的人类(或脚本)拿同样 2 小时去做穷举,Agent 就输了。\n\n## Agent 嘴上全会,手上只押 vLLM\n\nInferenceBench 最值得行业反思的结论,是它对 **Agent 行为轨迹(trace)的定性分析**:\n\n- **85.3% 的提交最终落到 vLLM 上**,SGLang、TGI 等其它引擎几乎被默认忽略。Agent 在有限预算里强烈收敛到「单一推理框架」,而不是去探索「哪个引擎更合适」。\n- Agent 在 transcript 里**反复点名**的优化技术:**Chunked prefill 99%、Quantization 100%、Speculative decoding 89%**—— 这些是它真的懂。\n- 但**实际非默认 vLLM launch 配置数很少**,Agent 把大量时间花在**反复重测、修复、超参微调**上,**而不是去探索截然不同的策略**。\n\n更扎心的是「**Found vs Submitted**」对比:Agent 经常在中间过程中「找到」一个看起来更优的配置,但因为没有正确验证 \u002F 持久化 \u002F 提交,**这个更好的版本在最终提交里消失了**。换句话说,Agent 的真正瓶颈**不是不知道怎么做**,而是**不会系统地比较、记录、保留自己已经验证过的最优解**。\n\n论文的失败模式分析里,把这种行为称作「**premature stop**」:Agent 看到一个已经 PASS 的 baseline,出于「再改有风险」的判断,直接把这个版本当作终态提交 —— 而不是把它当成「下一步继续优化的起点」。\n\n## Trajectory Viewer:每个 Agent 都被「拆给你看」\n\n为了让这件事不只是论文里的文字,官方站点提供了 **Trajectory Viewer**:每个 run 的工具调用被自动归类为 **inspecting \u002F installing \u002F launching \u002F evaluating \u002F debugging \u002F 实际优化** 六类 episode,你可以**逐回合回放**任一 Agent 的 trace,或者把它放到一张「行为地图」里和所有其他 run 对比。从可视化里能直接看到:大多数 run 都只在「inspect → 编辑 → 评估 → 调试」四步里打转,**很少推进到更难、更结构化的优化**(比如算子融合、调度策略、KV cache 重组)。\n\n同时,官方还配套了一个 **Cost vs Performance 图**:横轴是该 Agent 跑完 12 轮评估的 API 成本(对数刻度),纵轴是相对加速,**点越靠左上说明「便宜又能打」**。这等于把「InferenceBench 不是一个对工程成本无感的玩具」写进了产品形态里。\n\n## 截至 2026-08 的 leaderboard 现状\n\nv1.0.4(Aug 2026)的 leaderboard 已经在跑 **Claude Opus 5†、Claude Sonnet 5†、GPT-5.6 Sol Ultra†、Kimi K2.7 Code†、Kimi K3†、Grok 4.5\u002F4.6(Grok Build)、Claude Opus 4.7\u002F4.8、Claude Fable 5、Gemini 3.5 Flash、GLM-5.2 Max、GPT-5.5 High\u002FxHigh、Kimi K2.6** 等 15 个以上 Agent 配置。带 † 的运行使用的是 **strict prompt**,显式说明「不允许第三方预量化 checkpoint、不允许改评估 harness」,正是为了让分数反映真正的「开放域工程能力」,而不是「靠记忆拿分」。\n\n截至目前,**Claude Opus 5† 排第一**——它不是单一场景跑得最猛的,而是「每个场景都能稳定交出有效最终服务器」。换句话说,这次的赢家是**「重复性能」**,而不是「单次峰值」。\n\n## 这事对行业意味着什么\n\n把 LLM 推理优化丢给 AI Agent 自己做,这件事本身就在拷问一个更基础的问题:**「端到端 AI 工程」到底什么时候能闭环?** InferenceBench 给出的答案是:**会的部分在变多,但仍远没到能让 AI 自己替代 ML Systems Engineer 的程度**。\n\n具体几个信号:\n\n- **Agent 在「知识枚举」这一层已经满分**:chunked prefill、量化、投机解码,Agent 都能列、能写、能配。但**「枚举→验证→保留→提交」这条链上,任何一环掉链子,优化就消失了**。这是「工程纪律」问题,不是「模型能力」问题。\n- **单一框架的强先验**:85.3% 押 vLLM,说明 Agent 训练数据里 vLLM 的曝光率压倒性地高,这反过来会让其它引擎**在 Agent 时代的可见度持续下降**——一个值得关注的「**AI 工具同质化**」信号。\n- **「最终提交要 score」的设计非常狠**:它直接把 reward hacking 写进了评分规则,失败的提交回到 PyTorch 基线。这倒逼所有想刷榜的人,必须真正把优化跑稳,而不是堆跑分脚本。\n- **Cost vs Performance 图把工程成本摆到了台面上**:未来我们要谈「Agent 优化 LLM 推理」,就不能只谈「几倍加速」,还要谈「花了多少钱的 API 才换来这几倍」。\n\n一句话总结:**InferenceBench 不是告诉你 AI Agent 有多强,而是告诉你 AI Agent 现在的「工程化弱点」具体卡在哪一步**。这比一份漂亮的速度榜单有用得多。\n\n## 关键链接\n\n- 论文:https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.20468\n- ICML 2026 收录页:https:\u002F\u002Ficml.cc\u002Fvirtual\u002F2026\u002F67887\n- 官方 leaderboard 与轨迹可视化:https:\u002F\u002Finferencebench.ai\u002F\n- 代码仓库:https:\u002F\u002Fgithub.com\u002Faisa-group\u002FInferenceBench\n\n*原始研究 \u002F 引用提示:Yeon, Rank, Andriushchenko (2026). InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents. arXiv:2607.20468 \u002F ICML 2026.*","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.20468","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"7ac06d8e-b074-4147-abfc-ffaa4c6b8744","ai-efficiency",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"90c626a3-6a50-4e50-b10d-fc2c9d33dfbb","en","InferenceBench: 15 frontier agents optimize LLM inference","InferenceBench (arXiv:2607.20468, ICML 2026) drops the task of deploying and squeezing an OpenAI-compatible inference server into the hands of frontier coding agents. Across 15 agent configurations, 1 H100, and a 2-hour budget, agents hit up to 8.08× over a vanilla PyTorch baseline and ~4.05× over inference-engine defaults (vLLM et al.), but still trailed a simple hyperparameter search by up to 11.53×. 85.3% of submissions landed on vLLM — agents name chunked prefill (99%), quantization (100%), and speculative decoding (89%), but their search is shallow and they routinely fail to keep the best configuration they found. Code and leaderboard are open-sourced by Jehyeok Yeon and colleagues at ELLIS Institute Tübingen.","# InferenceBench: handing LLM inference optimization to AI Agents — and the not-so-pretty scorecard\n\n> Give a code-capable AI agent a single H100, a 2-hour clock, and a target LLM that needs to be served via an OpenAI-compatible endpoint. How far can it actually optimize? ICML 2026's InferenceBench delivers a refreshingly cold answer.\n\n## What this paper is actually about\n\nInferenceBench is an **open-ended benchmark** from **Jehyeok Yeon, Ben Rank, and Maksym Andriushchenko** at the **ELLIS Institute Tübingen \u002F Max Planck Institute for Intelligent Systems \u002F Tübingen AI Center**, accepted at **ICML 2026** (arXiv:2607.20468, submitted 2026-05-20). The official code and leaderboard are open-sourced at the GitHub repository `aisa-group\u002FInferenceBench`.\n\nThe core mechanic is simple and ruthless: **hand each agent a target LLM, one H100, a 2-hour wall-clock budget, and one of four isolated optimization scenarios — then grade only the final submitted server**. Submissions must pass a correctness check and an integrity audit against reward hacking. Any failed, unreachable, or regressed final submission is scored at the PyTorch baseline; intermediate results do not count.\n\n## Four scenarios, four distinct bottlenecks\n\nInferenceBench isolates the optimization target into four non-overlapping lanes so that what an agent \"wins\" actually means something:\n\n1. **Prefill Latency** — long-context prompts, measured by **time-to-first-token (TTFT)**.\n2. **Decode Latency** — long generations, measured per output token.\n3. **Throughput** — concurrent traffic across **burst \u002F Poisson \u002F constant-rate** profiles.\n4. **All-In-One** — a balanced scenario using the geometric mean of latency and throughput metrics.\n\nIn each scenario, the agent has to install an engine, set flags, run benchmarks, and decide what to keep — all without knowing the optimal configuration in advance. The benchmark explicitly forbids third-party pre-quantized checkpoints and modifying the evaluation harness, to ensure the score reflects real engineering rather than memorized recipes.\n\n## 15 frontier agents, one scorecard\n\nDrawing on the paper and the official leaderboard (v1.0.4 as of August 2026, which just added the Cost column and Grok 4.5 \u002F 4.6):\n\n- **vs. a vanilla PyTorch baseline**: agents reliably deliver **up to 8.08× speedup** — strong, considering they start from scratch.\n- **vs. inference-engine defaults** (vLLM, SGLang, TGI): agents also **beat defaults by roughly 4.05× on average**.\n- **vs. a simple hyperparameter search over the same engines under the same budget**: agents **trail by up to 11.53×**.\n\nForced-engine ablations (the agent must submit via one specified engine): **TGI-only 8.31×, SGLang-only 7.69×, vLLM-only 6.17×**, while the **non-agent per-scenario best search hits 14.30×** — still ahead of every agent configuration tested.\n\nThe takeaway is sharper than the headline numbers: **agents are \"good engineers\", not \"the best engineers\".** In two hours, they can beat a PyTorch baseline and beat engine defaults, but a focused hyperparameter sweep given the same budget still wins.\n\n## Agents know everything, submit one thing\n\nThe most important qualitative finding is about what agents *actually do* during the run:\n\n- **85.3% of submissions land on vLLM.** SGLang, TGI, and other engines are essentially ignored. Agents funnel hard toward a single inference framework instead of exploring which engine is actually best for the scenario.\n- In their own transcripts, agents repeatedly name the right techniques: **Chunked prefill 99%, Quantization 100%, Speculative decoding 89%** — they really do know the menu.\n- But the count of **distinct non-default vLLM launch configurations** per run is tiny. Agents spend most of their budget re-measuring, repairing, and tuning hyperparameters on a small set of configurations — rather than exploring substantially different strategies.\n\nThe \"**Found vs Submitted**\" chart is the painful one: agents often find a better-looking configuration mid-run, but they fail to validate it, preserve it, or commit it to the final submission. The better version evaporates. The bottleneck is **not domain knowledge — it is the discipline of comparing, recording, and keeping the best verified solution**.\n\nThe failure-mode analysis calls this \"**premature stop**\": an agent sees a passing baseline, decides \"any further change might break deployment\", and submits that baseline as the final answer — instead of treating it as the starting point for the next round.\n\n## Trajectory Viewer: every agent, broken down\n\nTo make this more than a paper claim, the official site ships a **Trajectory Viewer**: each run's tool calls are automatically grouped into **inspecting \u002F installing \u002F launching \u002F evaluating \u002F debugging \u002F optimizing** episodes. You can replay any run step by step, or project every run onto a shared \"behavior map\" to compare strategies at a glance. The visualization shows what the prose says: most runs circle through *inspect → edit → evaluate → debug*, rarely pushing into harder, more structural optimizations (operator fusion, scheduling strategies, KV-cache layout changes).\n\nThere's also a **Cost vs Performance** plot: API cost of the 12-run evaluation (log scale) on the x-axis, aggregate speedup on the y-axis. **Points in the upper-left are cheap and fast.** This pushes \"AI agent optimization\" out of the toy zone and into a real engineering budget conversation.\n\n## Leaderboard as of 2026-08\n\nThe v1.0.4 leaderboard tracks **Claude Opus 5†, Claude Sonnet 5†, GPT-5.6 Sol Ultra†, Kimi K2.7 Code†, Kimi K3†, Grok 4.5 \u002F 4.6 (Grok Build), Claude Opus 4.7 \u002F 4.8, Claude Fable 5, Gemini 3.5 Flash, GLM-5.2 Max, GPT-5.5 High \u002F xHigh, Kimi K2.6** — 15+ agent configurations. Runs marked with † use a **strict prompt** that explicitly forbids disallowed behaviors (no third-party pre-quantized checkpoints, no evaluation-harness edits) so that the score reflects genuine open-ended engineering rather than memorized recipes.\n\n**Claude Opus 5† currently ranks first** — not because it dominates any single scenario, but because it consistently delivers a valid final submission across runs. **Repeatability beats peak.** That's the leaderboard's defining message.\n\n## What this means for the industry\n\nHanding LLM inference optimization to AI agents is itself a stress test of a more basic question: **when does \"end-to-end AI engineering\" actually close the loop?** InferenceBench's answer: **more than yesterday, much less than the hype suggests.**\n\nA few signals worth tracking:\n\n- **The \"knowledge enumeration\" layer is already maxed out.** Chunked prefill, quantization, speculative decoding — agents can name, write, and configure all of them. But if any single link in the \"enumerate → validate → preserve → submit\" chain breaks, the optimization disappears. This is a **discipline** problem, not a **model** problem.\n- **Single-framework overfit.** 85.3% landing on vLLM is a strong prior signal: agent training data is dominated by vLLM exposure, which will keep reducing the visibility of other engines in the agent era. Watch this as an emerging **\"AI tool monoculture\"** risk.\n- **\"Final submission scores\" is a brutal design choice.** It bakes reward-hacking resistance into the scoring rules and forces anyone chasing the leaderboard to keep optimizations stable. No shortcuts.\n- **Cost vs Performance pulls engineering cost into the open.** Future discussions of \"AI agents optimizing LLM inference\" can't just talk about speedups — they have to talk about how much API spend it took to get there.\n\nOne-line takeaway: **InferenceBench doesn't tell you how strong AI agents are. It tells you exactly where their engineering weaknesses are right now.** That's a lot more useful than another pretty speedup leaderboard.\n\n## Key links\n\n- Paper: https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.20468\n- ICML 2026 listing: https:\u002F\u002Ficml.cc\u002Fvirtual\u002F2026\u002F67887\n- Official leaderboard + trajectory viewer: https:\u002F\u002Finferencebench.ai\u002F\n- Code repository: https:\u002F\u002Fgithub.com\u002Faisa-group\u002FInferenceBench\n\n*Original research \u002F citation hint: Yeon, Rank, Andriushchenko (2026). InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents. arXiv:2607.20468 \u002F ICML 2026.*","inferencebench-open-ended-llm-optimization","2026-08-16T12:00:00Z","2026-08-16T09:07:16.196159Z","2026-08-19T01:48:03.231362Z",true,"agent",127,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"ed39ed38-b5fa-4f58-92cf-d05233ab998b","Speculate with Memory：LLM Agent 无损加速 2.5×，准确率涨 39pp","speculate-with-memory-2-5x","2026-07-15T08:15:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"103c151a-3ac1-42b0-92d2-d2f910dbc6ea","大模型推理进入过思考时代：测试时计算的新问题","over-thinking-test-time-compute-paradox","2026-06-01T19:00:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"e07a88a3-bf96-4257-a54c-a7bcc705d5de","开源项目Forge：用Guardrails让8B模型Agent任务成功率从53%跃升至99%","forge-guardrails-8b-53-to-99-agent","2026-05-22T01:00:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"15747718-ff24-4026-9176-433bc5553bb7","NVIDIA AVO 刷满 ARC-AGI-3:Claude Opus 5 裸跑 30%,套上 agent 框架 100%","nvidia-avo-arc-agi-3-agent-harness","2026-08-23T15:30:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"0237222a-602b-47ef-9431-468009904428","FACET 先建环境再写任务:1.2K 轨迹把 Qwen3.5-27B 推到 Terminal-Bench 47.57,逼近 397B","facet-terminal-task-synthesis","2026-08-19T06:19:20+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"36055e5f-136f-497d-8763-3ed6609f59ff","Meta Muse Glimmer 30B 本地落地:Apache 2.0 的开源智能体,把 Agent 装进 24GB 显存","meta-muse-glimmer-30b-local-agent-apache2-r2","2026-08-19T03:00:00+00:00"]