Benchmark scores keep climbing, yet hand a frontier model a workbook inherited from a colleague and the picture changes fast. On September 30 the Surge AI team launched GDP.xlsx: 70 real workplace requests written by professionals, 20 configurations from 12 developers competing, and under all-criteria grading not one passes 40% of attempts. The paper, accepted to the Agent Evaluation Science Symposium in New York, hit arXiv on November 13.
How real the test is
All 70 requests were written by professionals across 13 domains, each starting from a workflow where frontier models had failed them, and every task passed three rounds of review. Inputs are 58 Excel workbooks plus 12 CSV files, about 2.18 million populated cells in total. Requests are short, with a median of 55.5 words, and 58 of the 70 never tell the agent which tab to check. The authors are Suhaas Garre, Sushant Mehta, and Edwin Chen of Surge AI.
The rule: every criterion or fail
Agents work in a container with a terminal and a file editor, Python with openpyxl and pandas, plus LibreOffice and Tesseract OCR; each attempt gets up to 200 turns and 60 minutes, and only the model API is reachable, so evidence must come from the file itself. Grading is not string matching: an OpenCode judge agent can open the file and run code, ruling on each of the 3-to-30 criteria per task, about 11 on average, and writing its reasoning before each verdict. Strict pass@1 demands every criterion; 39 tasks also pair completeness with exclusivity, so one extra item is a fail.
The leaderboard: ugly at the top
Per the paper and the official leaderboard: Gemini 4 Argon leads at 38.3% (task-sampling standard error of at most 5.8 points), Claude Opus 5.5 at 30.3%, Claude Sonnet 5.5 at 29.1%; no other configuration exceeds 25%, and the median across the 20 configurations is 19.3%, half the leader. The middle is crowded: GPT 6 Astra 22.9%, Muse Spark 1.3 22.6%, Grok 4.7 22.3%, GLM 5.3 21.4%, Qwen 3.8 Max 20.3%, Kimi K3 13.4%, and Mistral Large 3 at 0.0. Iteration works: Gemini 4 Argon sits 28 points above Gemini 3.1 Pro, and wherever the leaderboard holds two versions of a model, the newer scores higher.
Where it fails: evidence outside the cells
The shared thread in the failure analysis is answers that omit evidence the file actually holds. Muse Spark 1.3, on a rent-forecast task over a workbook with 15 tabs, 15,910 populated cells and 9,857 formulas, recalled 6 of 9 tenancies with precision 6 of 6, and still failed: the three it missed live on a BreakClauses tab, one hidden by an AutoFilter, and they carry roughly $14M of three-year forecast net rent. GPT 6 Astra delivered 3 of 8 housebound codes in a clinical data dictionary, and on a certification tracker claimed the workbook never says who handles a pending certification, while another tab records exactly that. Gemini 3.8 Flash missed a cabinet labeled only in a diagram pasted onto Sheet1, the +MDY50UC017 unit.
Structure is data: of the 58 workbooks, 42 use cell fill colors to carry meaning, 34 use non-default font colors, 21 embed images or shapes, and 17 hide sheets, rows, or columns; the median workbook carries four such features. The blind spot the paper names: loading a workbook into data frames keeps its values and drops formats, comments, and drawings, which is precisely where the evidence often sits.
So what
The failure analysis offers a plain lesson: a few lines of Python list every tab, comment, filter, and embedded image in a workbook, and all four failures come down to skipping that retrieval step. For anyone pointing AI at business spreadsheets, the paper is a bucket of cold water: even the leader passes fewer than two attempts in five. The benchmark itself guards against contamination, with three public tasks, an Apache 2.0 harness, 67 held-out tasks, and a canary string asking that it stay out of training data. Reasoning keeps improving; reading a workbook to the end is still a human job. Would you send it your receivables file first?
Refs: paper https://arxiv.org/abs/2611.17766 ; harness https://github.com/surge-ai/gdp-xlsx ; leaderboard https://surgehq.ai/benchmarks/gdp-xlsx