[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-coding-agent-scaffold-bottleneck":3,"news-related-abccd085-d910-4846-aae1-5f2136238f3b":36},{"id":4,"title":5,"summary":6,"content":6,"original_url":7,"source_id":8,"tags":9,"translations":23,"news_slug":29,"published_at":30,"created_at":31,"modified_at":32,"is_published":33,"publish_type":34,"image_url":13,"view_count":35},"abccd085-d910-4846-aae1-5f2136238f3b","别怪大模型:Coding Agent 真正的瓶颈在「脚手架」——Queen's 35 版本控制变量实证","Coding Agent 的质量回退,工程社区第一反应往往是「模型又退步了」。但 Queen's University 的 Sghaier 等人在 arXiv:2607.03691 中指出:这是误诊。\n\n这是第一项把「脚手架」从 LLM 中剥离出来做控制变量的纵向研究。之前所有 SWE-bench 类工作都「固定脚手架、换模型」,本文反向操作——固定同一个底层 LLM,只换 scaffolding,看 35 个连续版本会怎样。\n\n研究覆盖 Codex、Qwen Code、Gemini、OpenCode、OpenHands 五大开源脚手架,先看生态:平均发布速度超过 2 次\u002F天,几个月累积几千个 issue。然后深挖 Qwen Code CLI 的 35 个顺序版本,每个版本都在 50 个分层抽样的 SWE-bench Verified 任务上跑分,全程锁定底层模型不变。\n\n结论令人警醒:任务成功率与效率的波动,绝大部分能追溯到具体 PR、具体架构组件,而不是底层模型。即使基座完全没动,一次 prompt 模板调整、一次工具执行顺序改动、一次上下文管理重构,都可能让质量曲线出现 5–10pp 量级的跳变。\n\n这篇论文的价值不在新方法,而在打脸一个普遍盲区:业界把算力、参数、榜单都押在模型上,却没人系统地度量脚手架本身的演化贡献。它呼吁把脚手架当成「一等公民」来监控——Agentic QA、Scaffolding Observability 应该像对待模型一样有 SLA、有回归测试。\n\n对正在自研 Coding Agent 的团队,这意味着:升级脚手架前必须做「控制变量基准」,否则你们会陷入「升一版差 5%、再升一版回来」的玄学循环。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.03691","7437aeb9-930c-4866-a2e9-48003c1a792b",[10,14,17,20],{"id":11,"name":12,"slug":12,"description":13,"color":13},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",null,{"id":15,"name":16,"slug":16,"description":13,"color":13},"e82b2d09-81b2-43d1-977e-e018443b3c14","coding-agent",{"id":18,"name":19,"slug":19,"description":13,"color":13},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":21,"name":22,"slug":22,"description":13,"color":13},"c187600e-804c-4697-b828-1e4330e0eb10","qwen",[24],{"id":25,"lang":26,"title":27,"summary":28,"content":13},"9ac9c990-61bc-42f1-abbc-3ea1d5acfea3","en","Don't blame the model: scaffolding is the real bottleneck","The first reaction of the engineering community when Coding Agent quality regresses is usually \"the model has regressed again\". But Sghaier et al. of Queen's University point out in arXiv:2607.03691: this is a misdiagnosis. This is the first longitudinal study that strips \"scaffolding\" out of the LLM as a controlled variable. All previous SWE-bench-type work has \"fixed the scaffolding, swapped the model\"; this paper does the reverse — fix the same underlying LLM, only swap the scaffolding, and see what happens across 35 consecutive versions. The study covers five major open-source scaffolds: Codex, Qwen Code, Gemini, OpenCode, OpenHands. First, looking at the ecosystem: average release rate is more than 2 times\u002Fday, accumulating thousands of issues over a few months. Then drilling into the 35 sequential versions of Qwen Code CLI, with each version scored on 50 stratified-sampled SWE-bench Verified tasks, the underlying model locked throughout. The conclusion is sobering: most of the fluctuation in task success rate and efficiency can be traced to specific PRs and specific architecture components, not the underlying model. Even with the base model completely unchanged, a single prompt template adjustment, a tool execution order change, or a context management refactor can cause quality curves to jump 5–10pp. The value of this paper isn't a new method, but in slapping a common blind spot: the industry bets compute, parameters, and leaderboards on the model, yet no one systematically measures the evolution contribution of the scaffolding itself. It calls for treating scaffolding as a \"first-class citizen\" to monitor — Agentic QA, Scaffolding Observability should have SLAs and regression tests like models. For teams developing their own Coding Agent, this means: you must do \"controlled-variable benchmarks\" before upgrading scaffolding, otherwise you'll fall into the mystic cycle of \"upgrade one version, get 5% worse; upgrade again, come back\".","coding-agent-scaffold-bottleneck","2026-07-04T03:55:25Z","2026-07-12T00:12:11.884901Z","2026-08-19T02:08:40.142862Z",true,"agent",94,{"items":37},[38,43,48,53,58,63],{"id":39,"title":40,"news_slug":41,"published_at":42},"7ba15299-8bee-4039-8bc7-dbb58754b562","SWE-bench Science:最强 Claude Code 修科学代码也不及格,四类失败模式被拆解","swe-bench-science-benchmark","2026-08-21T13:00:00+00:00",{"id":44,"title":45,"news_slug":46,"published_at":47},"5b76d928-8896-4bf8-8edd-e195ecf0094a","Ornith-1.5自报跑分赢了Claude,独立复测翻车了","ornith-1-5-benchmark-reality-check","2026-08-20T17:10:00+00:00",{"id":49,"title":50,"news_slug":51,"published_at":52},"4c54e7dc-46f6-4d0e-86c9-278995cf0da8","Stanford PTXBench:让 LLM 裸写 H100\u002FB200 PTX kernel,没有一个模型全过关","ptxbench-llm-ptx-gpu-kernel-benchmark","2026-08-19T14:00:00+00:00",{"id":54,"title":55,"news_slug":56,"published_at":57},"e846f1c6-4644-4f84-a664-83ec82734210","Meta Muse Spark 1.2 与 Muse Code 把「1.2 → 编程」的推理效率推回前沿","meta-muse-spark-12-coding-agent-54-index","2026-08-05T08:00:00+00:00",{"id":59,"title":60,"news_slug":61,"published_at":62},"4c9f74d4-0252-4e86-8b6e-85d38788eea6","开源编程模型三选一:GLM-5.2、DeepSeek V4、Qwen3.6","glm-5-2-deepseek-v4-qwen-3-6-coding","2026-07-27T06:00:00+00:00",{"id":64,"title":65,"news_slug":66,"published_at":67},"9382e481-e16b-4925-a0d1-3b24cd8ba22a","OpenAI 复审自家推荐的 SWE-Bench Pro:731 道题里约三成是「坏题」,榜单狂欢该降温","openai-swe-bench-pro-audit","2026-07-13T00:11:00+00:00"]