Existing agent benchmarks share one premise: the task is well-specified — an issue report, a spec, a ticket. Real customer delivery is nothing like that. Requirements are ambiguous, key constraints are never written down, and the system is live. FDE-Gym, submitted to arXiv on October 15, targets exactly this gap and quickly climbed the Hugging Face Daily Papers board with 285 upvotes.

How the benchmark works

Built by a 26-author team at DeepLethe, FDE-Gym packs 95 end-to-end delivery tasks drawn from 20 customer scenarios across 18 sectors, each adapted by industry experts from real delivery projects spanning banking, healthcare, logistics, manufacturing and energy. The agent is placed in the role of a forward-deployed engineer: inside a simulated customer environment, it interacts with scripted stakeholders — where messages, trial runs and waiting time all carry penalties — and modifies a production-like system until it delivers a working solution.

Scoring ignores what the agent did and measures how much the customer's own number improved: normalized between 0 (system untouched) and 1 (expert reference solution). The GitHub repository (open source, Apache 2.0) phrases it more bluntly: on a 100-point scale, 80 or more counts as accepted, and an incident in production caps the score at zero.

Results: information is not the bottleneck

The team evaluated 13 model configurations from seven providers. In the realistic setting, where unstated constraints must be discovered, the best configuration achieves a mean score of just 0.56 and cleanly resolves only 14 of the 95 tasks.

Two findings deserve attention:

  • Discovering unstated constraints is the primary bottleneck. Per the authors, withholding those constraints costs leading models 0.28 on average; a misdirected request costs little more.
  • Knowing is not doing. On the 23 tasks where the best configuration obtained every key fact from stakeholders, it still reached a business uplift of 0.8 or above on only 10 of them.

Worse, weaker configurations often leave the customer worse off than doing nothing at all.

So what

This benchmark pours cold water on the current leaderboard race: high scores on ticket-style evaluations like SWE-bench say little about delivering in a live customer environment. For buyers, the questions FDE-Gym forces — score by the customer's KPI, charge for every interaction, one incident kills the run — are closer to what procurement should ask than any benchmark rank. For individual developers, the team is opening an online practice platform for humans on November 1, 2026, turning FDE skills into a trainable gym: once the agent writes the code, asking the right person, finding the real problem, and answering for what happens live become the human moat.

Reference: arXiv:2610.22350 (https://arxiv.org/abs/2610.22350)