The scientific community has spent decades accumulating executable knowledge — atmospheric models, ocean circulation, plasma simulation, quantum many-body computation — code that produces numbers matching the real world. Yet this knowledge never became a primary source of training experience for large models: toolchains are fragmented, domain conventions implicit, and correctness criteria specialized. A 45-researcher team led by PhAI Labs tore down that wall. Released September 16, ScienceIDE (arXiv:2609.19134) turns the world's scientific codebases into programmable, verifiable training environments for agents, and ranked #2 on Hugging Face Daily Papers the next day.

Where the scientific experience bottleneck lies

The paper names this problem the "scientific experience bottleneck": knowledge in scientific code is executable, yet cannot be directly converted into reliable learning experience. General coding benchmarks come with ready-made test suites, but the "correctness" of scientific code requires recompiling, re-running physics cases, and comparing against reference output. So the team built environments rather than datasets: guided by expert-defined scientific cases and acceptance criteria, agents transform the repositories themselves into programmable environments supporting task generation, execution, and scientific verification — one set of environments serving supervised fine-tuning, reinforcement learning, and evaluation at once.

Task mechanics: reward is "the simulation is right again"

Every task is a containerized episode on a pinned, unmodified upstream scientific codebase. Two task routes: Repair injects a semantic defect into the source, and the agent must find and fix it so the physics cases pass again; Implementation excises the body of a routine, and the agent must reimplement it so the solver reproduces the incumbent results. The scoring formula is deliberate: reward_repair = max(0, (reward − floor)/(1 − floor)), where floor is what the unfixed build already scores — an agent that changes nothing earns exactly zero. Reward can only come from "the simulation is numerically correct again", not from gaming a diff comparison.

The repository publishes 15 of 64 environments (Athena++, MITgcm, Gkeyll, DScribe, NEST, EDKit, Stim, and others), covering astrophysical MHD, ocean biogeochemistry, plasma kinetics, materials descriptors, and quantum many-body physics; the other 49 are named and held out as a test set. Of the 85 ScienceIDE-Hard tasks, 30 are published, each execution-validated: the official fix scores 1.0 in the grading container, the unfixed build leaves room for a reward signal, and the agent image contains no answer material.

What the models learned

Using verified interaction trajectories, the team trained PhAI-IDE-4B/9B/72B (now on Hugging Face). On scientific tasks: after SFT, the 9B model rose from 0.3125 to 0.5000 on the LAPS environment, and the 4B went from zero to 0.3333 on PLUTO dust particles; RL (starting from Qwen3.5-4B with async GRPO) pulled LAPS from 0.357 to 0.857 and MITgcm biogeochemistry from 0.286 to 0.571. More interesting is the positive transfer: the 4B gained 6.99 points on CodeXGLUE defect detection, and the 9B jumped from 27.20 to 63.20 on BBH Word Sorting — a full 36 points of spillover from scientific repair experience to general capability.

Gains are not uniform, though, and the paper itself reports a decline on HumanEvalFix Python for the 9B.

So what

The RL backend implements no trainer of its own — it runs on PSRL, a modified veRL, which tells you the center of gravity is environments, not training frameworks. The project is explicitly marked as a preview; the full environment set and task-authoring pipeline live in the ScienceInfra repository pending maturity. The real signal here: once you can give AI a physics-world judge that verifies correctness by itself, decades of dormant scientific code become a renewable training resource. For teams building vertical-domain agents, the sequence "acceptance criteria first, environments second, models last" is worth copying.

References: arXiv 2609.19134, github.com/aitofound/ScienceIDE