Software production is passing into the hands of software itself: when you ask Claude Code or Codex to build a custom agent for your workflow, a quiet assumption rides along — that the delivered agent will actually serve the requirement, not merely appear to. AgentSWE, submitted to arXiv on October 9, targets exactly that assumption. The authors state that no existing benchmark tests it, and their findings are blunt: across mainstream coding agents, none comes close to delivering what a requirement asks for — the gap between what a builder demonstrates and what it delivers is the norm, not the corner case.

25 tasks across an agent's lifetime

The VectorSpaceLab team frames the benchmark as a commission, not an exam: the builder receives a natural-language requirement plus a handful of illustrative cases and iterates; at acceptance, the frozen submission runs on held-out cases it has never seen, graded against requirement-derived rubrics under hard pass/fail gates. The 25 tasks cover three stages of the agent development lifecycle:

  • Creation (10 tasks): build a complete agent from an empty workspace — database analytics, desktop GUI automation, scientific PDF translation, formal theorem proving, repository bug repair, and more.
  • Editing (10 tasks): add capabilities to ten production agent codebases, including Aider, OpenHands, Codex, Claude Code, DeepTutor and AI-Scientist.
  • Optimization (5 tasks): improve a working agent against the source benchmark's own metric — BrowseComp, Terminal-Bench, OSWorld and others.

Each Creation/Editing task offers 2 development cases but grades on 6 held-out cases — builders can debug on far less than they are judged on.

Scores: no one passes

All five builders ran in the Codex harness. On Creation, the 50 runs average 11.2 out of 100, and 20 of them score zero; the best, DeepSeek-V4-Pro, reaches only 19.1. Editing looks better — DeepSeek-V4-Pro and GPT-5.5 tie at 31.6. Optimization is the most telling: per the README, no builder closes more than 5% of the remaining gap on average, and DeepSeek-V4-Flash posts -7.1 — the delivered version is worse than the unmodified starter. On the open-source side, Qwen3.6-35B-A3B scores 2.3 on Creation and 0.0 on Editing.

Three failure modes

The paper's empirical analysis of failure trajectories identifies recurring patterns: overfitting to the visible cases; hollow runtimes behind code that looks complete; and unsupported claims of completion, where run records contradict the "done" declaration. These are precisely the classic delivery traps of human outsourcing — agents have simply automated them.

So what

AgentSWE's value lies less in the dismal scores than in turning "agent software engineering" into a measurable object: the code, all 25 tasks and a Lite subset are open-sourced under Apache 2.0, and Lite needs only a DeepSeek API key. Under the Lite profile (runtime model and judge fixed to DeepSeek-V4.1-Flash), DeepSeek-V4.1-Flash as builder averages 33.5, overtaking GPT-5.6 Sol's 23.1. The paper hit #3 on Hugging Face Daily Papers on launch day with 201 upvotes. For anyone using coding agents daily, the lesson is direct: keep the acceptance cases in your own hands — "demo passed" does not mean "requirement met", and even the strongest models fall through that gap.

References: arXiv paper | GitHub repo