On June 12, 2026, a U.S. export control directive suspended access to Claude Fable 5 for 19 days, and Anthropic confirmed it in a statement the same day. Yining Hua, a researcher at Harvard University, treated the accident as a natural experiment: the people and the projects stayed the same, and only the model changed. The paper tracks the GitHub output of 297 Fable 5 users, and the conclusion stings: with or without a model that leads by 10.8 points on SWE-Bench Pro, about the same amount of work got done.

How the experiment was built

Model attribution relies on a Claude Code default: it adds a Co-Authored-By line naming the model to the commits and pull requests it helps create. The author collected these public PRs through the GitHub Search API and spot-checked 200 of them; the model named in the description matched the commits in 96% to 99% of cases. The treated group is 297 developers who used Fable 5 in the three days it was available; the control group is 8,908 developers who stayed on Opus 4.8. The estimate is a difference-in-differences model with developer and calendar-day fixed effects. By the two system cards cited in the paper, Fable 5 scores 80.0% on SWE-Bench Pro and Opus 4.8 scores 69.2%, a 10.8-point gap. The switch really happened: 82.3% of the treated group's PRs named Fable 5 before the suspension, 0.9% during it (74.6% named Opus 4.8), and only 26.9% after access returned.

Three flat numbers

When Fable 5 was available, the 7-day merge rate was only 1.4 percentage points higher than during the suspension (95% CI -1.4 to +4.3), not statistically significant; the placebo that compares the pre-release weeks with the suspension gives -2.6. Across five upgrades in the Opus and Sonnet families between February and September 2026 — 1,208 developers and 15,442 PRs — switching to the new model changed the merge rate by -0.9 points (-1.9 to +0.2); within the same developer and repository it was -1.2 (-2.3 to -0.02), significantly negative. In the six upgrades with published scores, the newer model was 5 to 18 points higher, and five of the six point estimates were negative. Weekly merged PRs moved +2.4% (-4.9% to +10.2%) and the share of PRs with failed automated checks moved +0.3 points; none of it was significant. Nobody slowed down during the ban either: PRs opened per developer per day went 4.8 before, 5.0 during, 5.2 after.

Why: a ceiling plus a task mismatch

In the week before the upgrades, 94.5% of these developers' PRs were already merged within 7 days, 77.1% within an hour, and 83.8% of them had every single PR merged — the merge rate was already at its ceiling. What a merge means matters more: among 2,019 sampled PRs, 95% were merged by the author and 90% had no approval from anyone else, with a median time to merge under half an hour. A merge says the author decided the work was done, not that anyone certified the quality. The author's hypothesis: the tasks developers hand to agents were already within the older model's ability. An internal Anthropic survey cited by the paper found that of 132 engineers, more than half could fully hand off only 0 to 20% of their work. The benchmarks are mismatched too: an independent March assessment from METR supports this — many PRs that pass SWE-bench would not be merged into main by maintainers at all.

The cold water

A merge rate is not productivity and not code quality; it is only the best public signal that a developer decided to keep a change. Only about 1% of merged Claude Code PRs name the model in the description, so the sample has selection bias, though refitting on all PRs pointed the same way. The 19-day window makes the intervals wide, but the five upgrades agree with much narrower ones. The upgrade analysis is observational: developers who pick new models by themselves may already be better at using them, which would bias the estimate upward — and even then, nothing rose. The paper is a single-author v1 preprint and has not been peer-reviewed. The lead is also benchmark-specific: on Terminal-Bench 2.1, Fable 5 led by only 1.6 points, largely because one in five of its runs fell back to Opus 4.8 after a safety refusal.

The conclusion in one line: benchmark gains of 5 to 18 points did not show up in the work developers actually kept. Next time a launch deck shows off a new score, it is worth asking first — will this number show up in my merge log?

(Paper: https://arxiv.org/abs/2611.23216 ; Anthropic statement: https://www.anthropic.com/news/fable-mythos-access ; METR note: https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/ )