A tax agent worked out both closing dates correctly and still failed to tell the client something: if the sale slipped by another three months, the exclusion worth up to $250,000 would vanish. That is not a calculation error. It is a missed question.

Surge AI put up DAYJOB: Tax on October 27. The benchmark contains 50 professional assignments written by practicing tax experts, each packaged with a full working environment full of client records. Scoring is pass@1: 50 tasks, 5 attempts per task, and an attempt only counts as a pass when it satisfies at least 95% of that task's rubric criteria. Claude Sonnet 5.5 leads with a 10.8% pass rate, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2% (full leaderboard).

What counts as a pass

The bar is hard in two separate ways.

First, the rubrics are long. The published dataset card puts the median number of rubric criteria per task at 30.5, and grading covers calculations, supporting evidence, consequential issues, and the finished deliverable (dataset card). Second, the threshold is high: a single attempt must hit at least 95% of the criteria to count. An agent that gets nearly everything right still scores zero if it drops one judgment call that would have changed the conclusion.

Tasks use the containerized Harbor format, with network access restricted to permitted model API hosts during execution — browsing and package downloads are blocked, so there is no looking things up on the fly. Only 2 of the 50 tasks are published; the full set has to be requested by email.

The scoreboard

On Surge AI's own run, top to bottom: Claude Sonnet 5.5 at 10.8%, Claude Fable 5.1 at 9.6%, Claude Opus 5.5 at 9.2%, Muse Wonder 1.0 at 6%, Grok 4.7 at 5%, GPT-6 Astra at 4.8%, GPT-6.1 Sol at 4%, Muse Spark 1.3 at 3.6%, Gemini 4 Argon at 3.2%, Hy4 Preview at 1.6%, DeepSeek V4.1 Flash at 1.2%, Gemini 3.8 Flash at 0.8%, GLM 5.3 at 0.4%, and Kimi K3, Mistral Large 4, and Qwen 3.8 Max all at zero.

This is the evaluator running its own questions, and no outside lab has reproduced the numbers yet. But one pattern is hard to miss: once professional work requires active judgment, the gap between models does not narrow — the entire tier drops below 10%.

Judgment, not arithmetic

Surge AI published two case studies in the release post (original).

Take the case of Anna, selling a condo. She moved out in August 2024 and wanted to compare closing in October 2026 against January 2027. The agent correctly concluded that either date satisfies the two-of-five-year residence requirement for the capital gains exclusion. What it missed is the other half: once the sale drifts past the end of August 2027, she stops qualifying. Across five attempts, Astra never flagged that deadline. Opus caught it in all five.

The second is a separating couple dividing a Roth IRA and a 401(k). Opus 5.5 based all five recommendations on the assumption that the couple were not legally married, telling them to keep the accounts separate with a cash payment to even out the balances. But they had lived together in Texas and held a ceremony in Colorado, and Texas recognizes informal marriages under certain conditions, while a valid marriage established elsewhere may remain valid in Florida. That distinction could change the tax consequences. Astra checked marital status first in four of five attempts.

The pattern across both cases is uncomfortable: the arithmetic is largely fine. What collapses is the part where you go looking for the issue the client never raised.

The files are the hard part

The 50 tasks share 1,789 files, with a median of 30 per task and as many as 123, including 1,487 PDFs, 117 Excel workbooks, 38 Word documents, and the rest emails, images, CSVs, and JSON records. There is deliberately non-standardized material in there — a photograph of a handwritten receipt for merchandise purchased for resale, where the agent has to read item descriptions and amounts and reconcile them against spending records; and an industrial silo-services company that its client assumes operates only in Alabama, while the field dispatch log shows crews working across five states.

These are not hard files. They are what real work actually looks like.

How this differs from earlier tax benchmarks

Existing tax benchmarks each focus on one slice. TaxBench asks 114 tax questions with defined answers. TaxCalcBench measures whether the numbers on a return are right. TaxLab tests whether an agent can question a simulated taxpayer, gather missing information, and complete a federal individual return. DAYJOB wants something broader: when the client asks the wrong question, can the agent still take the job on?

My read

The value of a benchmark is not the score. It is that it pushes the definition of "capability" forward by one notch. We used to ask whether an agent could do a task. Now someone is asking whether it can own a job end to end that nobody specified properly. That shift matters far more than the 10.8%.

Cold water is also due. The leaderboard is the evaluator's own run, only 2 sample tasks are public, the full set requires a request, and no third party can independently reproduce it today. Treat 10.8% as a directional signal, not a final verdict. And this benchmark covers one slice of one profession — reading it directly as "agents cannot do professional work" would be overreach.

The thing worth watching is what sits behind those 30.5 criteria. Models can already get a well-specified question right. The next skill to learn is spotting that the question is missing a piece. Continued growth in pretraining corpus will not fix that; only feedback carrying real consequences will, and consequence-bearing feedback is exactly the kind of data professional domains have the least of.