InferenceBench: handing LLM inference optimization to AI Agents — and the not-so-pretty scorecard

Give a code-capable AI agent a single H100, a 2-hour clock, and a target LLM that needs to be served via an OpenAI-compatible endpoint. How far can it actually optimize? ICML 2026's InferenceBench delivers a refreshingly cold answer.

What this paper is actually about

InferenceBench is an open-ended benchmark from Jehyeok Yeon, Ben Rank, and Maksym Andriushchenko at the ELLIS Institute Tübingen / Max Planck Institute for Intelligent Systems / Tübingen AI Center, accepted at ICML 2026 (arXiv:2607.20468, submitted 2026-05-20). The official code and leaderboard are open-sourced at the GitHub repository aisa-group/InferenceBench.

The core mechanic is simple and ruthless: hand each agent a target LLM, one H100, a 2-hour wall-clock budget, and one of four isolated optimization scenarios — then grade only the final submitted server. Submissions must pass a correctness check and an integrity audit against reward hacking. Any failed, unreachable, or regressed final submission is scored at the PyTorch baseline; intermediate results do not count.

Four scenarios, four distinct bottlenecks

InferenceBench isolates the optimization target into four non-overlapping lanes so that what an agent "wins" actually means something:

  1. Prefill Latency — long-context prompts, measured by time-to-first-token (TTFT).
  2. Decode Latency — long generations, measured per output token.
  3. Throughput — concurrent traffic across burst / Poisson / constant-rate profiles.
  4. All-In-One — a balanced scenario using the geometric mean of latency and throughput metrics.

In each scenario, the agent has to install an engine, set flags, run benchmarks, and decide what to keep — all without knowing the optimal configuration in advance. The benchmark explicitly forbids third-party pre-quantized checkpoints and modifying the evaluation harness, to ensure the score reflects real engineering rather than memorized recipes.

15 frontier agents, one scorecard

Drawing on the paper and the official leaderboard (v1.0.4 as of August 2026, which just added the Cost column and Grok 4.5 / 4.6):

  • vs. a vanilla PyTorch baseline: agents reliably deliver up to 8.08× speedup — strong, considering they start from scratch.
  • vs. inference-engine defaults (vLLM, SGLang, TGI): agents also beat defaults by roughly 4.05× on average.
  • vs. a simple hyperparameter search over the same engines under the same budget: agents trail by up to 11.53×.

Forced-engine ablations (the agent must submit via one specified engine): TGI-only 8.31×, SGLang-only 7.69×, vLLM-only 6.17×, while the non-agent per-scenario best search hits 14.30× — still ahead of every agent configuration tested.

The takeaway is sharper than the headline numbers: agents are "good engineers", not "the best engineers". In two hours, they can beat a PyTorch baseline and beat engine defaults, but a focused hyperparameter sweep given the same budget still wins.

Agents know everything, submit one thing

The most important qualitative finding is about what agents actually do during the run:

  • 85.3% of submissions land on vLLM. SGLang, TGI, and other engines are essentially ignored. Agents funnel hard toward a single inference framework instead of exploring which engine is actually best for the scenario.
  • In their own transcripts, agents repeatedly name the right techniques: Chunked prefill 99%, Quantization 100%, Speculative decoding 89% — they really do know the menu.
  • But the count of distinct non-default vLLM launch configurations per run is tiny. Agents spend most of their budget re-measuring, repairing, and tuning hyperparameters on a small set of configurations — rather than exploring substantially different strategies.

The "Found vs Submitted" chart is the painful one: agents often find a better-looking configuration mid-run, but they fail to validate it, preserve it, or commit it to the final submission. The better version evaporates. The bottleneck is not domain knowledge — it is the discipline of comparing, recording, and keeping the best verified solution.

The failure-mode analysis calls this "premature stop": an agent sees a passing baseline, decides "any further change might break deployment", and submits that baseline as the final answer — instead of treating it as the starting point for the next round.

Trajectory Viewer: every agent, broken down

To make this more than a paper claim, the official site ships a Trajectory Viewer: each run's tool calls are automatically grouped into inspecting / installing / launching / evaluating / debugging / optimizing episodes. You can replay any run step by step, or project every run onto a shared "behavior map" to compare strategies at a glance. The visualization shows what the prose says: most runs circle through inspect → edit → evaluate → debug, rarely pushing into harder, more structural optimizations (operator fusion, scheduling strategies, KV-cache layout changes).

There's also a Cost vs Performance plot: API cost of the 12-run evaluation (log scale) on the x-axis, aggregate speedup on the y-axis. Points in the upper-left are cheap and fast. This pushes "AI agent optimization" out of the toy zone and into a real engineering budget conversation.

Leaderboard as of 2026-08

The v1.0.4 leaderboard tracks Claude Opus 5†, Claude Sonnet 5†, GPT-5.6 Sol Ultra†, Kimi K2.7 Code†, Kimi K3†, Grok 4.5 / 4.6 (Grok Build), Claude Opus 4.7 / 4.8, Claude Fable 5, Gemini 3.5 Flash, GLM-5.2 Max, GPT-5.5 High / xHigh, Kimi K2.6 — 15+ agent configurations. Runs marked with † use a strict prompt that explicitly forbids disallowed behaviors (no third-party pre-quantized checkpoints, no evaluation-harness edits) so that the score reflects genuine open-ended engineering rather than memorized recipes.

Claude Opus 5† currently ranks first — not because it dominates any single scenario, but because it consistently delivers a valid final submission across runs. Repeatability beats peak. That's the leaderboard's defining message.

What this means for the industry

Handing LLM inference optimization to AI agents is itself a stress test of a more basic question: when does "end-to-end AI engineering" actually close the loop? InferenceBench's answer: more than yesterday, much less than the hype suggests.

A few signals worth tracking:

  • The "knowledge enumeration" layer is already maxed out. Chunked prefill, quantization, speculative decoding — agents can name, write, and configure all of them. But if any single link in the "enumerate → validate → preserve → submit" chain breaks, the optimization disappears. This is a discipline problem, not a model problem.
  • Single-framework overfit. 85.3% landing on vLLM is a strong prior signal: agent training data is dominated by vLLM exposure, which will keep reducing the visibility of other engines in the agent era. Watch this as an emerging "AI tool monoculture" risk.
  • "Final submission scores" is a brutal design choice. It bakes reward-hacking resistance into the scoring rules and forces anyone chasing the leaderboard to keep optimizations stable. No shortcuts.
  • Cost vs Performance pulls engineering cost into the open. Future discussions of "AI agents optimizing LLM inference" can't just talk about speedups — they have to talk about how much API spend it took to get there.

One-line takeaway: InferenceBench doesn't tell you how strong AI agents are. It tells you exactly where their engineering weaknesses are right now. That's a lot more useful than another pretty speedup leaderboard.

Key links

Original research / citation hint: Yeon, Rank, Andriushchenko (2026). InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents. arXiv:2607.20468 / ICML 2026.