arXiv 2606.19704 introduces a new framework for LLM Agent evaluation, developed by IBM and 14 partner organizations. The framework replaces the "average score" metric with "predictive validity" — i.e., how well a benchmark score predicts real-world task success.
The problem with average scores: most LLM Agent benchmarks report an "average score" across a set of tasks. But average score is a poor predictor of real-world success — a model that scores 80% on the benchmark might fail on 50% of real-world tasks, because the benchmark tasks don't capture the full distribution of real-world complexity.
The "predictive validity" framework: the authors propose that a benchmark should be evaluated by its "predictive validity" — the correlation between the benchmark score and the real-world task success rate. They define a "validity coefficient" that captures this correlation, and they show that most current benchmarks have low validity (r=0.3-0.5) when correlated with real-world task success.
The "validity-improved" benchmark: the authors release a new benchmark — AgentEval-Valid — that has been specifically designed for high predictive validity. The benchmark includes 1,200 tasks drawn from real customer deployments, with a diversity that matches real-world task distributions. Models that score high on AgentEval-Valid are 2-3× more likely to succeed on real-world deployments.
The bigger takeaway: "predictive validity" should be the new standard for LLM Agent evaluation. The "average score" approach is misleading, and the industry needs benchmarks that actually predict real-world success. For the industry, this means Agent vendors should report "predictive validity" alongside "average score," and enterprises should use validity-validated benchmarks for model selection.