Software increasingly functions as part of the scientific instrument itself — observation, analysis, and simulation all run on code underneath. That means a bug in scientific code can damage not just program behavior, but the very evidence behind published conclusions. Yet mainstream coding-agent evaluations mostly report aggregate task-success scores, and rarely answer the more important question: when an agent fails to repair scientific software, how does it fail?
SWE-bench Science, submitted to arXiv on August 20 by the OpenMOSS team, goes straight at that gap (paper).
A Benchmark That Targets Scientific Software
First, the composition: 119 tasks drawn from 98 GitHub repositories across 20 scientific domains, all repository-level real-world repair scenarios. Each task falls into one of three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration.
The results are brutal. The best-performing agent is Claude Code with Opus-5 (max), and its pass@1 still sits below 50%. In other words, put today's highest-scoring coding agents in front of real scientific-software repairs, and more than half the tasks don't get done.
Where It Goes Wrong: Four Recurring Failure Mechanisms
The failure analysis is more valuable than the headline score. The paper identifies four recurring failure modes:
- Deficits in scientific knowledge or abstraction — lacking the domain knowledge or abstraction ability required;
- Misguided exploration or surface-level repair — exploring in the wrong direction, or stopping at superficial fixes;
- Incomplete repair coverage or system integration failure — patches that don't cover the full scope, or breakdowns at integration;
- Failure to generalize — scientific knowledge that doesn't transfer beyond the observed cases.
These aren't just flavors of "not capable enough" — they point to four different research directions: some need knowledge, some need better exploration strategy, some need different integration approaches. Aggregate scores tell you how bad; failure mechanisms tell you where to fix.
The Counterintuitive Part: Feeding Scientific Knowledge Isn't Always Helpful
The paper also runs a paired ablation: removing explicit scientific guidance while preserving the repository and executable engineering context. The conclusion — scientific knowledge is not uniformly beneficial:
- Well-grounded, aligned information constrains the repair and improves average performance and token efficiency;
- Poorly aligned guidance induces anchoring, pinning the model to a wrong direction, without necessarily improving exact repair success.
That's a warning for the whole RAG and prompt-engineering community: stuffing the wrong "authoritative information" into context is worse than stuffing in nothing.
Why This Paper Deserves a Careful Read
Three personal observations.
First, general-purpose coding benchmarks are saturating fast, and "can fix a generic repo" is a different question from "can fix a scientific repo". What makes scientific software special is that the code is part of the chain of evidence — repair requires domain understanding, not just pattern matching. This benchmark draws that boundary clearly.
Second, the failure-mechanism taxonomy has more long-term value than the pass@1 number. It decomposes the vague impression "agents aren't good enough" into concrete, researchable, improvable objects — which is exactly what a benchmark should do.
Third, the ablation finding that knowledge injection is not uniformly beneficial directly challenges the engineering intuition that more context is better. Wrong guidance is more dangerous than no guidance, because anchoring makes the model dig deeper in the wrong direction.
So What?
Next time you see a headline about a coding agent setting a new record, ask one more question: what's its pass@1 on scientific code? In an era where software is the instrument, that number may be closer to answering "can AI genuinely participate in science" than any general leaderboard.