Training resumed. The loss was nonzero, the parameters kept updating, the curve kept moving. By any conventional acceptance check, the agent had finished the job.

A paper (arXiv 2611.14364, from Shanghai Artificial Intelligence Laboratory, Fudan University, East China Normal University, Shanghai Jiao Tong University, Renmin University of China and Peking University, posted on November 10) puts the distance between "running again" and "repaired correctly" in plain sight. Its CylindFormer case is an image-to-point-cloud registration training job that had to be brought back to life. Two separate executions both satisfy the same operational conditions: training resumes, loss is nonzero, parameters update. One passes all 20 checklist items. The other passes 16. That second run quietly changed coordinate handling, overlap labels and matching losses - all of which the reference paper in the workspace had specified. The authors flag the boundary themselves: the comparison concerns method consistency during recovery, and differences in final registration accuracy still require separate measurement.

What the benchmark actually is

The tasks are not written from scratch. They are cut out of real work. The authors recruited actual researchers and used agent sessions from their own projects as the skeleton: every task anchors on one genuine user request, compresses the still-effective earlier requests into a Background section, and replays the complete workspace to that moment without cleaning it up.

The 75 tasks span seven research stages - Survey, Design, Data & Env., Code, Execution, Analysis and Writing - and map onto 33 requirements and 150 sub-requirements out of a full taxonomy of 207. They compile into 1,173 binary checklist items, 15.6 per task on average.

The replayed workspace holds 41.1 files on average, and 65.3% of tasks contain files the agent itself wrote in earlier turns. A block of 36.0 earlier user turns is compressed into a state summary that cites 10.3 still-effective earlier requests on average. Each task averages 2.5 deliverables, with instructions 1,677 characters long.

One ruler, two scores

Seven frontier models, three rollouts per task, 225 task runs in total, with verdicts issued by a GLM-5.3 judge. On instance success rate - every applicable checklist item must pass - GPT-6.1-Sol reaches 66.22%, Claude Opus 5.5 56.89%, DeepSeek-V4.1-Flash 39.56%, Qwen3.8-Max 37.78%, GLM-5.3-Flash 30.67%, Kimi-K3 28.00%, and Qwen3.8-27B 25.78%. Roughly one third of the leader's runs still failed to satisfy every requirement.

The sharper result is Table 8, where the deliveries and the item-level verdicts are held completely fixed and only the definition of what counts changes. DeepSeek-V4.1-Flash scores 69.33% when only delivery contracts and general norms are checked; adding historical requirements and material-or-workspace requirements drops it to 40.00%, and the full checklist puts it at 39.56%. GLM-5.3-Flash goes 69.33% to 31.56% to 30.67%. Kimi-K3 goes 69.78% to 28.44% to 28.00%. The gap between DeepSeek and Kimi widens from 0.44 points to 11.56 points.

Correctness is what blocks delivery

Broken out by dimension, correctness is the lowest-scoring dimension for all seven models. Correctness instance success rate ranges from 31.11% to 71.11%, against 72.00%-87.56% for completeness and 79.11%-93.78% for constraint compliance. Between 85.5% and 92.8% of failed runs contain at least one unmet correctness item. DeepSeek-V4.1-Flash reaches 93.16% completeness CSR but only 47.11% correctness ISR, and 39.56% overall.

The stage spread is just as pronounced: Kimi-K3 ranks third on Code at 73.96%, yet its Execution and Writing scores are 47.47% and 47.31%, the latter last of all seven models. Design and Execution are the hardest stages for every model.

More expensive is not more reliable

Under the paper's frozen prices, GPT-6.1-Sol costs about USD 2.17 per run and Claude Opus 5.5 about USD 5.69 - roughly 2.62 times as much for an instance success rate 9.33 points lower. The cheap options do not win either: GLM-5.3-Flash gets 30.67% at roughly USD 0.20 per run, DeepSeek-V4.1-Flash 39.56% at roughly USD 0.53. Context length is no better a predictor: Claude Opus 5.5 peaks at 104.71K and Qwen3.8-27B at 108.02K, close together, yet their instance success rates differ by 31.11 points.

Three caveats the authors raise themselves

The limitations section is unusually blunt. The harnesses are not uniform: GPT-6.1-Sol runs on Codex while the other six run on Claude Code, so cross-model differences reflect the model and the harness together. The judging is not purely objective either - the rubric is still written by humans, judge disagreements concentrate in one direction and mostly reorder adjacent models, and independent human annotations are listed as the next step. The sample is not necessarily representative: source sessions may over-represent particular projects, tools and researcher populations, cut-point sampling does not guarantee coverage of all research workflows, and replay plus dependency recovery can lose context that the original researcher could see. Separating the contribution of session context from that of workspace state, the paper says, needs future ablations.

The authors also bound the whole result: the scores should be read as evidence about this released task distribution rather than as a universal conclusion. Worth noting, the introduction opens with Qwen's own reported 93.0% for Qwen3.8-Max on PaperBench Code-Dev as a counterexample - a variant that does not check execution, and does not check whether the paper's empirical results were reproduced.

So what

The real contribution here is not another score. It is the separation of two questions that usually get merged: whether the agent moved the project forward, and whether the agent moved the project off the rails. A complete set of deliverables does not mean the method was not quietly swapped out. In the CylindFormer runs, training came back up in both cases - and the difference lands between 20/20 and 16/20.

Paper: https://arxiv.org/abs/2611.14364 ; project page: https://arebench.com