The bill for evaluating agents is spiraling. The EarlyEval paper puts it plainly: a single pass of a frontier model over an agentic benchmark can cost hundreds to thousands of dollars — a price paid repeatedly across iterative development cycles. The Shanghai Jiao Tong University team, whose report landed on arXiv on Sep 2, offers a different fix: don't drop tasks, don't switch to a smaller model — just halt the agent run the moment its outcome becomes predictable.
A new axis for cheaper evaluation
The mainstream route to cheaper evaluation so far has been benchmark distillation: compress hundreds of tasks into fewer, which shrinks the task count but leaves the cost of each retained task untouched. EarlyEval attacks a complementary axis — early stopping inside each task. Its core observation is simple: an agent's final outcome is often evident from its intermediate behavior well before execution completes; a doomed run shows it in the first few steps.
Method-wise, the framework trains a pair of LightGBM classifiers (one for success, one for failure) over behavioral, textual, and reference-solution features. The moment either classifier crosses a calibrated confidence threshold, the run is halted. The per-step overhead of this watch, the paper notes, is negligible.
The numbers: how much saved, at what distortion
Across three benchmarks — SWE-bench Verified, TerminalBench, and Toolathlon — the paper reports EarlyEval eliminates 13%-26% of agent steps and up to 44.1% of input tokens and 29.4% of output tokens, at 89%-97% prediction accuracy, while perturbing per-agent resolve rates by only one to two percentage points on average.
That trade deserves a closer look: for high-frequency internal evaluation, swapping 1-2 points of leaderboard noise for nearly half the input tokens is almost certainly a win; for public leaderboard submissions whose numbers must survive audit, the perturbation has to be handled with care. Notably, the paper reports resolve-rate impact as a headline metric rather than burying it in an appendix.
The repo: all code, no artifacts
The companion GitHub repository (inphotoo/earlyeval) is a code-only release: training, testing, feature construction, ablation, architecture comparison, policy replay, and table-generation code are all there, including the 4x4 model-by-harness cross-diagnosis scripts for TerminalBench. Trained models, prediction files, and paper tables are deliberately excluded — you rebuild them via the README's reproduction pipeline. For architecture comparison, the repo ships baseline code for direct MLP, BERT/CodeBERT, local LLM-logit, and Qwen fine-tuning alongside the main LightGBM model (I_LightGBM_Dense_AF). The output paths in the README contain an icse_submission_draft segment, hinting this is the companion code for an ICSE submission.
So what
The economics of agent evaluation is becoming its own discipline: training costs money, inference costs money, and now verifying whether any of it worked is priced in dollars too. EarlyEval's real contribution is not the 44.1% figure — it is turning evaluation budget from a constant into a dial: stop early when you are iterating, run to completion when you need the exact number. For teams burning cash on SWE-bench daily, this may be the most practical kind of research. The paper gathered 110 upvotes on Hugging Face Daily Papers on its first day (a single community board).
Reference: arXiv:2609.02783 — https://arxiv.org/abs/2609.02783