Nearly every vendor now advertises "deep research" ability: hand the agent a vague question and it searches the web, reads papers, and returns a report. But one question is hard to dodge: how do you know it actually got the right answer?
Static retrieval benchmarks have an easy answer: the answer is in the corpus, can you find it. The real web is not like that. The evidence for one question might sit half in page text, half in video subtitles, and half inside a virtual museum tour nobody tells the agent to look at.
Making evidence acquisition itself the exam
The paper introduces the OpenDS (Open-Web Deep Search) task and builds the OpenDS-World benchmark on top of it: 200 tasks spanning eight domains, each independently human-verified so that answers are unique. The complexity of the open web is broken into three categories: diverse modalities (text, images, audio, video), heterogeneous formats (HTML, PDF, MP3, MP4, Parquet), and interactive access requirements — some evidence can only be obtained by paging, entering parameters, or driving a graphical interface.
Best score 39.0%, and still 122 wrong
Twelve frontier models were evaluated under a unified tool environment. GPT-6 Astra took first place at 39.0% (78 of 200 tasks correct), followed by Claude Opus 5.5 at 38.0% and Gemini 3.8 Flash at 36.5%; Qwen3.8-Flash at 12.0% and DeepSeek-V4-Pro at 12.5% came last. Even the winner got 122 tasks wrong.
Difficulty hurts more than the headline number suggests. Averaged across the 12 models, accuracy falls from 39.66% on low-difficulty tasks to 23.50% on medium and 16.76% on high. GPT-6 Astra alone drops from 59.26% to 40.98% to 24.71%. On the hardest tier, Gemini 3.8 Flash leads at 28.24% while Qwen3.8-Flash manages only 4.71%.
Humans at 21.1%, below the top model
Across 1,000 independent solving attempts, human participants reached 21.1% overall accuracy, below the best-performing model. And for 42.5% of the tasks, no assigned human participant solved them at all.
That is not the same as saying models have surpassed human researchers — the tool conditions and time budgets on both sides may not be comparable. But it does show that when a question demands multi-hop evidence gathering across multiple sources and modalities, humans fail at scale too.
More tokens does not mean more accuracy
GPT-6 Astra averaged 51.05 tool calls and 18.66 minutes per task. GPT-5.6 Sol made nearly the same number of calls (52.42) but finished in 8.99 minutes, scoring 11 points lower. Gemini 3.8 Flash consumed roughly 5.22 times GPT-6 Astra's tokens and scored lower still. Tool composition explains just as little: Claude Opus 5.5 and Qwen3.8-Flash spend near-identical shares of calls on GUI work (51.52% versus 49.42%), yet they score 38.0% and 12.0%.
Failure is what actually costs money
Failed tool calls account for only 2.83%–9.42% of all calls, yet consume 19.27%–61.90% of total tool execution time. For GPT-6 Astra the split is 6.34% of calls against 42.57% of the time.
The budget is a one-hour wall-clock limit plus a 100-call limit. Exhaustion ranges from 16.5% for Claude Opus 5.5 to 81.5% for Qwen3.8-Flash. After a failure, Claude, Gemini, and GPT-5.6 Sol switch to a different tool roughly 79%–83% of the time, while Kimi K3 and GLM-5.3 retry with modified arguments about 37%–38% of the time. One group changes lanes; the other spins in place.
So what
The value of this evaluation (arXiv:2610.30181) is not the 39.0%. It is that it takes "an agent doing deep research" apart into three measurable things: picking the right sources, coordinating different tools, and stopping before the budget runs out.
The first two set the ceiling. The third sets the cost curve. An agent that averages 18 minutes and 50 tool calls, and one that runs 45 minutes without an answer, should not be priced as the same product.
The benchmark does not settle whether AI can do deep research for us. It leaves a sharper question: when your agent hits a dead end, is it changing tools, or just editing the same arguments?