[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-harnessdev-llm-selfbuilt-agent-harness":3,"news-related-14a7f5ab-e270-461c-b862-4bde139e463f":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"14a7f5ab-e270-461c-b862-4bde139e463f","HarnessDev 基准:让 LLM 自建 Agent Harness,代码领域仍输人类工程师","HarnessDev 把评估对象从 Agent 的任务输出换成它自建的执行基建:模型从最小种子搭出完整 harness 并迭代演化。6 模型、4 领域、2207 实例测试显示:写作与 ML 实验已追平人类参考,代码与搜索仍明显落后,演化收益不稳定、跨模型迁移有限。","Agent 圈这两年有个共识越来越硬:同一个模型,换一套执行框架,任务表现能差出一大截。模型权重之外的这层基础设施——planning、执行、记忆、验证的胶水代码,社区管它叫 agent harness。可现有的评测几乎都只报告\"某个选定 harness 之下\"的下游分数,模型自己会不会搭 harness、搭得好不好,长期是个盲区。字节跳动 Seed 团队参与的新基准 HarnessDev 就是冲着这个盲区去的,论文已提交 arXiv。\n\n## 评测对象换成了\"可运行的基建\"\n\nHarnessDev 的思路是把评估单元从任务输出整体搬到可运行的基础设施上,分两个阶段考察。Creation 阶段,agent 从一个最小种子和少量样例出发,要求搭出一套完整的执行系统;Evolution 阶段,则让它以自己刚搭好的 harness 为起点,根据下游执行反馈反复修改,目标是把基准成绩继续往上推。每一套构建出来的 harness,都在两个维度上打分:能力(留出基准上的任务成功率)和效率(执行消耗的 token 成本)。Creation 结果覆盖六个创作者模型、四个领域、五个下游基准,共 2207 个唯一下游实例,评估任务对开发过程全程隐藏。\n\n## 三条泼冷水的发现\n\n第一条,领域分化非常明显。生成的 harness 在代码和搜索研究领域仍明显落后于成熟的人类工程参考实现;但在写作和机器学习实验这两个领域,已经能追平甚至超过所选参考——顺带一提,不同模型生成的 harness 在执行成本上差异巨大。\n\n第二条,自我演化没那么神。Evolution 阶段确实能带来一些性能提升,但提升不稳定,而且只能部分迁移到留出任务上,越接近真实泛化越露馅。\n\n第三条,harness 收益是\"模型绑定\"的。固定 runtime 模型的对照实验显示,一套 harness 带来的增益强烈依赖于究竟由哪个模型来执行它——换模型,收益就缩水,跨模型迁移性有限。\n\n## 为什么这篇值得读\n\n最近\"模型自建工具链\"的叙事很热,各家风向都在讲 agent 自己写 harness、自己进化的故事。HarnessDev 的价值恰恰在于给出了一个可复现的负面答案:在代码这类硬核工程领域,模型自建的执行系统离人类成熟实现还有肉眼可见的差距,而且优化出来的收益既不稳、也不通用。这提醒从业者,harness 工程短期内仍是人的活儿,所谓\"自进化 agent\"的宣传要打个折扣看。\n\n对研究者来说,它还示范了一种新的评测视角:与其在固定 harness 下刷任务分,不如把 harness 本身变成被测对象——毕竟当模型能力逐渐趋同,基建层的差异反而成了胜负手。\n\n所以,下次看到\"我们的 agent 会自己搭自己的运行环境\",先问一句:在 held-out 任务上、换个模型跑,还能保持这个提升吗?\n\n参考:arxiv.org\u002Fabs\u002F2609.01437","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.01437","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"e82b2d09-81b2-43d1-977e-e018443b3c14","coding-agent",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"51e182e5-d871-469f-96b5-2274e2608824","en","HarnessDev: LLM-Built Agent Harnesses Still Trail Human Code","HarnessDev: LLM-built agent harnesses match human references in writing and ML, still lag in code; evolution gains transfer poorly across models.","A quiet consensus has hardened across the agent ecosystem: run the same model under a different execution framework and task performance swings wildly. The glue layer outside model weights — planning, execution, memory, verification — is what the community calls the agent harness. Yet nearly every agent evaluation reports downstream scores under one chosen harness, leaving a model's ability to build the harness itself largely unmeasured. HarnessDev, a new benchmark from a team that includes ByteDance Seed researchers, targets exactly that blind spot; the paper is now on arXiv.\n\n## The unit of evaluation becomes runnable infrastructure\n\nHarnessDev shifts what gets measured from task outputs to runnable infrastructure, in two stages. In Creation, an agent starts from a minimal seed plus a handful of cases and must assemble a complete execution system. In Evolution, it takes its own finished harness and iteratively revises it using downstream execution feedback, aiming to push benchmark scores higher. Each constructed harness is scored on two axes: capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results span six creator LLMs, four domains, and five downstream benchmarks — 2,207 unique downstream instances in total, with evaluation tasks hidden throughout development.\n\n## Three cold-shower findings\n\nFirst, the split by domain is stark. Generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation — with large variation in execution cost across models.\n\nSecond, self-evolution is oversold. Evolution does produce some performance gains, but they are unstable and transfer only partially to held-out tasks.\n\nThird, harness gains are model-bound. Experiments with a fixed runtime model show the gains depend strongly on which model executes the harness — indicating limited transfer across models.\n\n## Why this paper matters\n\nThe narrative of agents building their own toolchains is running hot right now. The value of this benchmark is a reproducible negative answer: in hard engineering domains like code, model-built execution systems remain visibly behind mature human implementations, and the optimized gains are neither stable nor general. For practitioners, harness engineering stays human work for now — discount the self-evolving-agent marketing accordingly.\n\nFor researchers, it also demonstrates a fresh evaluation lens: instead of grinding task scores under a fixed harness, make the harness itself the object under test — as model capabilities converge, infrastructure differences become the deciding factor.\n\nSo the next time someone claims their agent builds its own runtime, ask one question: does the gain survive held-out tasks and a different model? (Source: arxiv.org\u002Fabs\u002F2609.01437)","harnessdev-llm-selfbuilt-agent-harness","2026-09-03T19:10:00Z","2026-09-03T19:09:47.349307Z","2026-09-03T19:09:47.349315Z",true,"agent",64,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"63c30bcd-3ffc-47c5-bd74-c2a9ed8f7c94","DeepSeek Harness 预览版开源:Agent 被拆成可插拔的插件栈,模型只负责想、Harness 负责做事","deepseek-harness-plugin-stack","2026-09-05T06:00:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"c77381d9-29ba-45ff-89df-855d11f90de2","Terminal-Universe:Qwen 把旧轨迹反向重建为 3.73 万个环境,27B 微调双基准 +11.9\u002F+13.8 分","terminal-universe-trajectory-environments","2026-09-04T17:10:00+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"00346b75-f071-42fd-ae16-db4c5569f01a","EarlyEval 提前叫停注定失败的 Agent:近半 token 省下,分辨率只动一两个点","earlyeval-early-stop-agent-eval","2026-09-03T21:04:52+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"089195fb-7fe5-4ba9-a4bc-8e356fe5e923","BAAI把1000个GitHub仓库蒸馏成5000个技能,科研agent奖牌率31%冲到73%","baai-disco-repo-to-skill-library","2026-09-03T17:07:35+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"7ba15299-8bee-4039-8bc7-dbb58754b562","SWE-bench Science:最强 Claude Code 修科学代码也不及格,四类失败模式被拆解","swe-bench-science-benchmark","2026-08-21T13:00:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"5b76d928-8896-4bf8-8edd-e195ecf0094a","Ornith-1.5自报跑分赢了Claude,独立复测翻车了","ornith-1-5-benchmark-reality-check","2026-08-20T17:10:00+00:00"]