On August 31, 2026, an arXiv paper (2608.27831) landed that exists specifically to challenge how coding agents are evaluated. A team at Sungkyunkwan University raised a question few had quantified head-on: coding agents are almost universally benchmarked on the SWE-bench family, whose tasks are built from curated GitHub issues — long, structured, and information-rich. Real user requests, however, are typically far shorter and more casual. The distribution gap between the two had never been systematically measured.
The Gap: 88 Percent Versus 7 Percent
The paper measures first and builds a benchmark second. The team defined a six-category information taxonomy and four dimensions of linguistic style, then applied both to two corpora: real user prompts from SWE-chat, and problem statements from SWE-bench Verified and Pro. The results are lopsided. Requests carrying only a problem statement — alone or with limited additional context — account for 88 percent of real prompts but just 7 percent of benchmark problems. Meanwhile, 87 percent of real prompts are casually written, whereas 94 percent of benchmark problems are formal. In other words, the benchmark exam assumes the candidate receives a fully documented brief, while in reality nearly nine in ten users just toss over a vague complaint.
381 Task Families, Rewriting the Same Problem N Ways
Guided by that measurement, the team built RealSWE: 381 multi-variant task families derived from SWE-bench Verified and Pro. Variants within each family share the same underlying task and the same gold patch, differing only in information composition and linguistic style — some variants add desired behavior, motivation, environment information, and reproduction steps one by one; others rewrite formal text into casual phrasing. This design turns how the input is written into a controlled variable: same model, same reference answer, the only thing changing is how the user asks.
Seven Frontier Models, Down 6.4 Points on Average
Evaluating seven contemporary LLMs with RealSWE, the team reports three findings. First, realistic inputs reduce resolution rates by 6.4 percentage points on average, and can change model rankings — the leaderboard order of today may not survive contact with the real input distribution. Second, controlled analysis shows that including Desired Behavior and Motivation significantly affects performance, whereas Environment Information and Reproduction Steps merely add tokens without measurable benefit. Third, linguistic style has only small, model-dependent effects — what really matters is not casual versus formal tone, but whether the information is present.
The So-What for Ordinary Users
The most valuable part of this paper is a directly actionable tip it leaves for ordinary users: most real prompts omit Desired Behavior and Motivation, and explicitly stating them substantially improves the software engineering performance of the model. Next time you brief a coding agent, instead of pasting a screenful of error logs, think first about one sentence — what exact behavior you want, and why. The accompanying RealSWE-bench data is open-sourced on GitHub under an MIT license; the evaluation harness has not been released yet, with the team saying it will arrive soon; see arXiv:2608.27831 (arxiv.org/abs/2608.27831). As benchmark tasks drift further from real usage, this paper is the first to hand us a ruler for the gap between topping the leaderboard and actually working.