[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-realswe-casual-prompts-coding-benchmark":3,"topics-all":35,"news-related-781a010e-a14f-415b-ae79-c3e7d6e9a815":54},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":21,"news_slug":28,"published_at":29,"created_at":30,"modified_at":31,"is_published":32,"publish_type":33,"image_url":14,"view_count":34},"781a010e-a14f-415b-ae79-c3e7d6e9a815","RealSWE 基准实测:用户随手一句话,7 个前沿编码模型掉分 6.4 个百分点","成均馆大学团队对比真实用户 prompt 与 SWE-bench 任务发现:88% 真实请求只带问题描述,基准任务里仅 7% 如此;87% 真实请求口语化,94% 基准任务书面化。据此构建 381 个多变体任务族,7 个前沿模型平均掉 6.4 个百分点,排名也会变。","2026 年 8 月 31 日,arXiv 出现一篇专门「挑刺」编码评测的论文(编号 2608.27831)。成均馆大学的研究团队提出一个此前很少有人正面量化的问题:如今编码智能体几乎都在 SWE-bench 系列上比拼分数,但这些任务来自精心整理的 GitHub issue——长、结构化、信息富集;而真实用户发给模型的请求,往往短得多、随意得多。这两者之间的分布差距,此前没有被系统测量过。\n\n## 差距有多大:88% 对 7%\n\n论文先做测量,再做基准。团队定义了一套六类信息分类法和四个语言风格维度,把它们同时套在两组语料上:一组是 SWE-chat 里的真实用户 prompt,另一组是 SWE-bench Verified 和 Pro 的问题陈述。结果相当悬殊:只带问题描述(或外加有限上下文)的请求,在真实用户 prompt 里占 88%,但在基准任务里只占 7%;反过来,87% 的真实 prompt 是口语化写法,而 94% 的基准任务用的是正式书面语。换句话说,基准考试默认考生会拿到一份信息齐全的卷子,而现实里近九成的用户只会丢过来一句含糊的抱怨。\n\n## 381 个任务族,把「同一道题」改写 N 遍\n\n基于这个测量,团队构建了 RealSWE:381 个多变体任务族,全部派生自 SWE-bench Verified 和 Pro。每个任务族内部的变体共享同一个底层任务和同一个 gold patch,只在信息构成和语言风格上不同——有的变体把期望行为、动机、环境信息、复现步骤一样样加上去,有的变体把书面语改写成口语。这种设计让「输入怎么写」变成了一个可控变量:同一个模型、同一份参考答案,唯一变化的是用户怎么提问。\n\n## 七个前沿模型,平均掉 6.4 个百分点\n\n用 RealSWE 评测七个当代大模型后,团队得到三个结论。其一,真实风格的输入让解决率平均下降 6.4 个百分点,而且足以改变模型之间的排名——今天榜单上的座次,放到真实输入分布下未必成立。其二,受控分析显示,在 prompt 里写明「期望行为」和「动机」会显著影响性能;相比之下,环境信息和复现步骤只是白白增加 token,没有可测量的收益。其三,语言风格本身的影响很小,且依模型而异——真正要命的不是口语还是书面语,而是信息是否到位。\n\n## 对普通用户的「所以呢」\n\n这篇论文最有价值的部分,恰恰是它给普通用户留了一句可直接执行的提示:大多数真实 prompt 都漏掉了「期望行为」和「动机」这两样东西,而把它们明确写出来,能实质性地提升模型的表现。下次给编码智能体提需求时,与其贴一整屏报错日志,不如先想清楚一句话——你要的到底是什么行为、为什么要它。论文配套的 RealSWE-bench 数据已在 GitHub 以 MIT 许可开源,评测 harness 尚未放出,团队表示将很快释出;详见 arXiv:2608.27831(arxiv.org\u002Fabs\u002F2608.27831)。当基准任务和真实使用越来越脱节,「刷榜」和「能用」之间的那道缝,这篇论文第一次给出了尺子。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.27831","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18],{"id":12,"name":13,"slug":13,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"e82b2d09-81b2-43d1-977e-e018443b3c14","coding-agent",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[22],{"id":23,"lang":24,"title":25,"summary":26,"content":27},"e3331b41-8b7b-4488-a5dd-26e7ece8a709","en","RealSWE: Casual Prompts Cost Coding Models 6.4 Points","RealSWE rebuilds SWE-bench into 381 variant families with casual user prompts; seven frontier models drop 6.4 points and rankings shift.","On August 31, 2026, an arXiv paper (2608.27831) landed that exists specifically to challenge how coding agents are evaluated. A team at Sungkyunkwan University raised a question few had quantified head-on: coding agents are almost universally benchmarked on the SWE-bench family, whose tasks are built from curated GitHub issues — long, structured, and information-rich. Real user requests, however, are typically far shorter and more casual. The distribution gap between the two had never been systematically measured.\n\n## The Gap: 88 Percent Versus 7 Percent\n\nThe paper measures first and builds a benchmark second. The team defined a six-category information taxonomy and four dimensions of linguistic style, then applied both to two corpora: real user prompts from SWE-chat, and problem statements from SWE-bench Verified and Pro. The results are lopsided. Requests carrying only a problem statement — alone or with limited additional context — account for 88 percent of real prompts but just 7 percent of benchmark problems. Meanwhile, 87 percent of real prompts are casually written, whereas 94 percent of benchmark problems are formal. In other words, the benchmark exam assumes the candidate receives a fully documented brief, while in reality nearly nine in ten users just toss over a vague complaint.\n\n## 381 Task Families, Rewriting the Same Problem N Ways\n\nGuided by that measurement, the team built RealSWE: 381 multi-variant task families derived from SWE-bench Verified and Pro. Variants within each family share the same underlying task and the same gold patch, differing only in information composition and linguistic style — some variants add desired behavior, motivation, environment information, and reproduction steps one by one; others rewrite formal text into casual phrasing. This design turns how the input is written into a controlled variable: same model, same reference answer, the only thing changing is how the user asks.\n\n## Seven Frontier Models, Down 6.4 Points on Average\n\nEvaluating seven contemporary LLMs with RealSWE, the team reports three findings. First, realistic inputs reduce resolution rates by 6.4 percentage points on average, and can change model rankings — the leaderboard order of today may not survive contact with the real input distribution. Second, controlled analysis shows that including Desired Behavior and Motivation significantly affects performance, whereas Environment Information and Reproduction Steps merely add tokens without measurable benefit. Third, linguistic style has only small, model-dependent effects — what really matters is not casual versus formal tone, but whether the information is present.\n\n## The So-What for Ordinary Users\n\nThe most valuable part of this paper is a directly actionable tip it leaves for ordinary users: most real prompts omit Desired Behavior and Motivation, and explicitly stating them substantially improves the software engineering performance of the model. Next time you brief a coding agent, instead of pasting a screenful of error logs, think first about one sentence — what exact behavior you want, and why. The accompanying RealSWE-bench data is open-sourced on GitHub under an MIT license; the evaluation harness has not been released yet, with the team saying it will arrive soon; see arXiv:2608.27831 (arxiv.org\u002Fabs\u002F2608.27831). As benchmark tasks drift further from real usage, this paper is the first to hand us a ruler for the gap between topping the leaderboard and actually working.","realswe-casual-prompts-coding-benchmark","2026-09-06T17:06:25Z","2026-09-06T17:07:59.064731Z","2026-09-06T17:07:59.064741Z",true,"agent",135,[36,45],{"slug":37,"tag_slug":37,"title_zh":38,"title_en":39,"intro_zh":40,"intro_en":41,"id":42,"is_active":32,"created_at":43,"modified_at":44},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":46,"tag_slug":46,"title_zh":47,"title_en":48,"intro_zh":49,"intro_en":50,"id":51,"is_active":32,"created_at":52,"modified_at":53},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":55},[56,61,66,71,76,81],{"id":57,"title":58,"news_slug":59,"published_at":60},"14a7f5ab-e270-461c-b862-4bde139e463f","HarnessDev 基准:让 LLM 自建 Agent Harness,代码领域仍输人类工程师","harnessdev-llm-selfbuilt-agent-harness","2026-09-03T19:10:00+00:00",{"id":62,"title":63,"news_slug":64,"published_at":65},"7ba15299-8bee-4039-8bc7-dbb58754b562","SWE-bench Science:最强 Claude Code 修科学代码也不及格,四类失败模式被拆解","swe-bench-science-benchmark","2026-08-21T13:00:00+00:00",{"id":67,"title":68,"news_slug":69,"published_at":70},"5b76d928-8896-4bf8-8edd-e195ecf0094a","Ornith-1.5自报跑分赢了Claude,独立复测翻车了","ornith-1-5-benchmark-reality-check","2026-08-20T17:10:00+00:00",{"id":72,"title":73,"news_slug":74,"published_at":75},"4c54e7dc-46f6-4d0e-86c9-278995cf0da8","Stanford PTXBench:让 LLM 裸写 H100\u002FB200 PTX kernel,没有一个模型全过关","ptxbench-llm-ptx-gpu-kernel-benchmark","2026-08-19T14:00:00+00:00",{"id":77,"title":78,"news_slug":79,"published_at":80},"e846f1c6-4644-4f84-a664-83ec82734210","Meta Muse Spark 1.2 与 Muse Code 把「1.2 → 编程」的推理效率推回前沿","meta-muse-spark-12-coding-agent-54-index","2026-08-05T08:00:00+00:00",{"id":82,"title":83,"news_slug":84,"published_at":85},"9382e481-e16b-4925-a0d1-3b24cd8ba22a","OpenAI 复审自家推荐的 SWE-Bench Pro:731 道题里约三成是「坏题」,榜单狂欢该降温","openai-swe-bench-pro-audit","2026-07-13T00:11:00+00:00"]