A single benchmark produced two numbers 43.5 percentage points apart. Co-Design Bench drew 273 real "revise this app" tasks out of 26,314 actual user coding sessions and ran ten coding-agent endpoints against them. The best score in the authors' own evaluation: 69.9% of acceptance criteria passed on average, but only 26.4% of tasks fully delivered. In other words, more than seven in ten tasks still carried at least one requirement the agent never met. Full paper
What a real request looks like
The tasks here were not written from scratch. The pipeline starts from 39,325 production queries across 26,314 real coding sessions, deduplicates them down to 37,463, then ranks candidates by difficulty, verifiability and historical failure signals before consolidating them into 273 bounded tasks. Every starting application must build, launch and render, or it gets pruned outright.
The suite splits into two tracks. Multimodal Editing (ME) holds 100 tasks with screenshot annotations and optional image, audio, video or document attachments, carrying 553 criteria in total. Repository Editing (RE) holds 173 tasks requiring coordinated changes across components, routes, state and data flow, carrying 1,778 criteria. Together that is 2,331 hidden expert-authored acceptance criteria, invisible to the agent during execution. The benchmark needs no gold implementation and no target screenshot.
The leader in the authors' evaluation, gpt-6-astra, scores 0.754 strict completion on Multimodal Editing with a 41.0% full-delivery rate. On Repository Editing, the same endpoint drops to 0.667 strict and 17.9% full delivery. Identical scoring, different track, roughly half the number.
Grading follows a judge-then-refute protocol. An evaluator grades each criterion Pass, Partial or Fail using source diffs, rendered screenshots and browser interaction traces. Then a separate LLM verifier challenges every positive verdict, hunting for stubs, hardcoded outputs and unreachable entry points. Two frontend experts additionally hand-checked the rubric scores for 200 task-model outputs.
The authors also stress-tested the grader itself. Rescoring two solvers with three alternative judges shifted absolute graded scores by at most 0.053, and the ME-versus-RE ordering flip between kimi-k3 and Qwen3.8-27B held under all four evaluator configurations. The measurement looks stable even though the judge is an LLM.
Three things more interesting than the leaderboard
First, the track changes the ranking. Strict-completion rankings correlate at Spearman ρ = 0.818 across the two tracks, which looks solid until you look closer: kimi-k3 drops from fourth on ME to sixth on RE, while Qwen3.8-27B climbs from fifth to second. The spread between endpoint means is 28.7 points on ME but only 13.0 on RE, so a single combined leaderboard hides setting-specific strengths.
Second, doing more work is not the same as doing better. On ME, glm-5.2 averages 69 turns and burns 7.29M tokens for a strict score of 0.572. gpt-6-astra uses 32 turns and 1.24M tokens and reaches 0.754. On RE, Qwen3.8-27B spends 57 turns and 6.72M tokens to score 0.645, still losing to gpt-6-astra's 0.667 at 48 turns and 2.33M tokens. The paper's phrasing is that higher completion does not require uniformly greater execution effort.
Third, the broader the edit, the harder the task. Tasks spanning one or two change types score 0.654 in strict completion; three or four types drop to 0.591. The zero-pass rate rises from 0.9% to 5.8%. Change more places, and more gets missed.
Fourth, one counter-intuitive detail. Broken down by attachment modality, document tasks carry the highest strict score at 0.652, yet their full-delivery rate is only 18.2%, below the 37.0% of text-only tasks. The input modality with the best criterion coverage is exactly the hardest one to deliver completely.
Agents are assembling their own toolchain
Another signal hides in the tool calls. OpenCV appears in 68.5% of video task runs, and ffmpeg in 31.5%. Among audio runs, 25.0% call ffmpeg and 4.4% call faster-whisper. More directly, agents reach out to Hugging Face and pull down pretrained perception models themselves: AST, CLAP, Whisper, Qwen2-Audio, CLIP, DINOv2 and OWLv2 all show up in the trajectories.
The frontier model serves as the brain while OpenCV and Whisper become the eyes and ears. That route may be closer than building ever-larger natively multimodal models.
There is also a controlled comparison worth noting: on the same WebCompass-edit tasks under the same scoring rubric, switching qwen3.7-plus from single-shot generation to an agentic loop lifts mean graded completion from 73.6% to 95.1%, and the count of all-pass tasks from 49 to 104. Changing the execution mode pays off faster than changing the model.
My read
Reporting a single average score is the form most likely to mislead: it collapses "touched every requirement a little" and "actually finished the job" into the same bucket. This benchmark puts both numbers side by side. 69.9% against 26.4% is a 2.6x gap, and that gap tracks real productivity far better than any ranking does.
One caveat: every number above comes from the authors' own evaluation, and there is no third-party replication yet. The released dataset covers only the 96 ME tasks.
So the next time an agent benchmark hands you one average score, ask: how many were actually finished?