A quiet consensus has hardened across the agent ecosystem: run the same model under a different execution framework and task performance swings wildly. The glue layer outside model weights — planning, execution, memory, verification — is what the community calls the agent harness. Yet nearly every agent evaluation reports downstream scores under one chosen harness, leaving a model's ability to build the harness itself largely unmeasured. HarnessDev, a new benchmark from a team that includes ByteDance Seed researchers, targets exactly that blind spot; the paper is now on arXiv.
The unit of evaluation becomes runnable infrastructure
HarnessDev shifts what gets measured from task outputs to runnable infrastructure, in two stages. In Creation, an agent starts from a minimal seed plus a handful of cases and must assemble a complete execution system. In Evolution, it takes its own finished harness and iteratively revises it using downstream execution feedback, aiming to push benchmark scores higher. Each constructed harness is scored on two axes: capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results span six creator LLMs, four domains, and five downstream benchmarks — 2,207 unique downstream instances in total, with evaluation tasks hidden throughout development.
Three cold-shower findings
First, the split by domain is stark. Generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation — with large variation in execution cost across models.
Second, self-evolution is oversold. Evolution does produce some performance gains, but they are unstable and transfer only partially to held-out tasks.
Third, harness gains are model-bound. Experiments with a fixed runtime model show the gains depend strongly on which model executes the harness — indicating limited transfer across models.
Why this paper matters
The narrative of agents building their own toolchains is running hot right now. The value of this benchmark is a reproducible negative answer: in hard engineering domains like code, model-built execution systems remain visibly behind mature human implementations, and the optimized gains are neither stable nor general. For practitioners, harness engineering stays human work for now — discount the self-evolving-agent marketing accordingly.
For researchers, it also demonstrates a fresh evaluation lens: instead of grinding task scores under a fixed harness, make the harness itself the object under test — as model capabilities converge, infrastructure differences become the deciding factor.
So the next time someone claims their agent builds its own runtime, ask one question: does the gain survive held-out tasks and a different model? (Source: arxiv.org/abs/2609.01437)