Human researchers take for granted something AI scientists never had: a world to work in. Centuries of literature sit there — readable, citable, but not runnable. The methods described in papers cannot be executed, so an AI can only read research on static text, not do it. Scientist World, released by Taiming Lu, Sachin Konan and Zhuang Liu, fills in that missing floor: it turns arXiv papers one by one into executable research environments — 1,000 in the first release, built entirely by a multi-agent pipeline with no human reviewing any paper at any stage.

Building the world: 124,816 papers screened into 1,000 labs

Selection starts from 124,816 unique papers across arXiv's core AI categories over fourteen months. Two fast screens with MiniMax M2.7 keep roughly the top 25% quality band and empirical-method papers, leaving 21,354; detailed judgments on research quality and reproduction difficulty follow, and a geometric-mean utility picks a category-balanced top 1,000. The source adds about 9,000 papers a month, of which about 1,500 survive — the world keeps growing with the literature.

The build is a fixed nine-iteration workflow: three reproducers, three designers, one packager, and two verifier passes, with the model family rotating across GPT, Claude, MiniMax and GLM to avoid a single builder's bias. Median cost is 0.4 GPU-hours per paper, at most 70; the 1,000 papers together consumed 2,286 GPU-hours, 239,506 compute jobs, 947,056 agent turns, and 51B input tokens.

Quality is not a promise but a measured property: in 99% of environments the reproduced method's score lands within a small margin of the paper's reported results; 82% beat the paper's own baselines; 37% of iteration milestones were reverted by later cross-checks — 573 times in total; and every environment must pass a verifier that took no part in building it. All 1,000 did.

The exam: 16 frontier models beat the authors 74% of the time

A typical environment carries 3.3 experiments, 32 metrics (16 visible, 16 hidden), and 59 files of about 3.8k lines of code, spread over eight areas. Half of every evaluation stays hidden from the contestant, so the score reflects generalization rather than tuning against the visible half. Scoring uses the authors' Frontier-Normalized Improvement (FNI): zero matches the reproduced paper method, negative is worse, and a frontier cohort's mean score is anchored near 100.

Sixteen models from seven providers entered: Anthropic's Fable 5.1/5 and Opus 4.8/4.7, four OpenAI models from GPT-6 Astra down to GPT-5.4, Zhipu's GLM 5.2/5.1, Moonshot's Kimi K2.7/K2.6, MiniMax M3/M2.7, Xiaomi's MiMo V2.5 Pro, and DeepSeek V4, entering through scaffolds such as Claude Code and Codex. The result: 84% of improve attempts and 74% of innovate attempts beat the paper's reproduced method — in most environments, the AI scientist does better than what the human authors published. On the leaderboard, Fable 5 is the strongest at improving existing methods, Fable 5.1 and Fable 5 take the top two spots at designing new ones, and among open models GLM 5.2 comes closest to the frontier, placing third at innovating.

Nothing was thrown away: the average attempt spans 5.4 hours, 6 method revisions, 25 experiments and 35 commits, all preserved as version-controlled repositories — roughly 6,400 hours of complete research traces.

The gym: a small model's ideas scale 2.6x

The world can also be used as a training ground. The authors trained Qwen3-8B with Tinker (LoRA rank 32): supervised fine-tuning on 1,000 trajectory examples, then GRPO over 1,500 already-scored ideas. Because an 8B model cannot yet carry long-horizon implementation, ideation was separated from execution — the trained model only proposes ideas, a single fixed implementer (GPT 5.6 Sol) executes every idea under identical conditions on ten held-out papers, so the idea's origin is the only variable. Median FNI rises from 41 (base) to 84 (SFT) to 106 (GRPO) — about 2.6x, the ↑158% in the abstract — while the mean climbs from 51 to 90, reaching the anchor of the frontier cohort.

So what

The hard part of this work is not the 74% headline; it is turning "can do research" from an adjective into a measurable engineering problem: reproduction margins, hidden halves, independent verification, FNI normalization — every layer is anti-gaming. The 37% reversion rate matters too: trust in a fully automated pipeline comes from mutual error-checking, not from faith. As model capability iterates monthly, the bottleneck is moving from "is the brain smart enough" to "is the world big enough." The literature already exists; what was missing was the pipeline that turns it into runnable environments — and that pipeline now has a monthly intake of about 1,500 more papers. What an AI scientist lacks was never intelligence; it was a world that lets it fail, repeatedly.

Reference: Scientist World: Grow a Research World for AI-Scientists