The race for scientific agents has reached the point of "doing a PhD student's job," yet a credible exam has been missing: existing benchmarks test either question answering or coding in isolation, not the full research workflow. On October 23, the Institute of Software at the Chinese Academy of Sciences, together with Alibaba's Qwen team, Zhejiang University, HKUST and Tsinghua, released SCI-AGENT-EXAM on arXiv — a 120-task unified exam for ten frontier models (paper).

The Exam: 156 Experts, 7 Research Task Types

The construction is unusually rigorous. A total of 156 domain experts participated, with task production and evaluation handled by independent groups. Every task was designed, built and verified by a PhD-level expert in the relevant domain, and the authors deliberately injected plausible distractors and misleading intermediate signals — bait for agents looking for shortcuts.

Tasks span 11 scientific domains, from physics, mathematics, materials and chemistry to bioinformatics and scientific computation, and are organized by the Scientific Workflow Task Taxonomy (SWTT) into seven types: Literature Synthesis, Hypothesis Validation, Tool Orchestration, Code Generation, Error Diagnosis, End-to-End Analysis, and Result Interpretation. Each task requires three deliverables — a structured final answer, executable artifacts (code and scripts), and textual artifacts (reports and explanations) — all executed under control inside a dedicated Docker container with the same command-line harness.

Scoring is dual-track: executable artifacts are re-run and verified by deterministic programs on hidden data, while textual artifacts go to a calibrated, expert-validated LLM judge. The two scores are reported separately before being combined.

The Leaderboard: Astra Far Ahead, Three Chinese Models in the Top Ten

The spread is wide. GPT-6 Astra ranks first at 80.9, ahead of second-place Claude Fable 5 by a full 13.6 points; Claude Opus 5 sits third at 64.2; Qwen3.8 Max takes fourth at 62.7 and Kimi K3 fifth at 62.6; GPT-5.6 Sol (59.6), Claude Opus 4.8 (57.1), GLM-5.3 (55.9) and Grok-4.6 (52.4) follow, with DeepSeek-V4-Pro last at 50.9.

Note the mismatch between programmatic and judge scores: Qwen3.8 Max scores only 54.7 on programmatic checks but 74.2 from the LLM judge — mediocre code, sparkling reports. The gap runs through the whole board.

Three Real Findings

The pass-rate paradox. Astra's 80.9 average looks great, but under the strict "single-run combined score ≥0.95" bar, only 26.0% of its runs qualify; Fable 5 manages just 9.7%. Averages hide variance: today's scientific agents are "good on average" and far from "reliable every time."

Strong reports ≠ correct code. The "strong report but weak execution" (TL) fraction ranges from 0.14 to 0.48 across models. DeepSeek-V4-Pro posts the highest TL rate (0.48) and the lowest dual-pass rate (TR, 0.25) — the archetypal smooth talker — while Astra is the most balanced with TR 0.83 and TL 0.14. Trajectory analysis adds that most models recover programmatic checks more reliably than scientific correctness: getting a result to run is easier than getting the science right.

Expensive yet cheaper. Astra costs $20.79 per run for the top score — a lower total cost than Qwen3.8 Max or Kimi K3. GLM-5.3 is the most expensive at $57.96 per run, averaging 102.4 turns. The paper's conclusion is blunt: evaluate efficiency at the run level, not by token price.

The Specialist Profiles

Astra leads in 10 of the 11 domains, but others have both strengths and blind spots: Claude Fable 5 scores 80 in Engineering yet 49 in Bioinformatics; Kimi K3 hits 81 in Materials Science but 50 in Physical Science. On task types, Qwen3.8 Max, GLM-5.3 and Kimi K3 do relatively best on Literature Synthesis — Chinese models first find their footing at the "reading papers" stage.

So What

The exam itself matters more than the ranking: it decomposes the full research workflow into 120 executable, verifiable tasks rather than another answer sheet. Two caveats: the leaderboard was produced on the Claude Code harness, and conclusions may not transfer to other harnesses; and the paper's code repository link still returns 404 at the time of writing, so replication waits for the artifacts. While every vendor tells an "AI scientist" story, the 26% pass rate is the reminder: agents that can reliably carry a PhD-level workflow are not yet within reach.