Early October, Solidot relayed an Ars Technica story that laid bare what people inside engineering orgs had been suspecting: AI coding tools may be more productive at producing code than at producing software. A Harvard working paper by economists Fiona Chen and James Stratton stitched Jellyfish engineering telemetry to GitHub activity signals across more than 700 software companies, around 700,000 developers, and roughly 300 million work events from 2021 through March 2026, asking a single question: did companies ship more software after AI coding agents landed?

The numbers are blunt: code up, output unchanged

Once firms adopted AI coding agents — Claude Code, Cursor, Devin, and friends — total lines of code rose about 30 percent on average, commits 20 percent, pull requests 23 percent. By every traditional developer-productivity metric, that should be a win.

But the same dataset shows Jira issue resolution and epic completion did not move in any statistically significant way. Writing code got cheaper. The shipped feature did not.

The bottleneck did not disappear — it moved downstream. Time from PR submission to merge climbed by roughly 49 percent. Comments per pull request climbed roughly 35 percent. Reversion and follow-up fixes nearly doubled. By March 2026, 80 percent of firms had AI code-review tools live, yet only 23 percent of review comments came from an AI; the rest, and the longer hours, still landed on humans.

Three details worth dwelling on

First, writing got cheaper; reviewing got expensive. AI coding agents use 5 to 30 times more tokens than a chatbot for an equivalent task. That cost shows up as commit volume, while the comment churn, the rebases, and the back-and-forth show up on engineers’ timesheets. Gartner’s August warning about agent inference costs rising fivefold inside two years is not abstract — it is exactly what a 49 percent merge-time blow-up looks like when nobody re-pools the workflow.

Second, Jira is closer to truth than GitHub. Issues and epics are tied to business outcomes. Commits and PRs are artefacts of process. The authors’ central claim is that the latter rises while the former does not, and the disconnect gets worse when managers reward the latter as "velocity".

Third, AI code review tooling is far less adopted than the marketing implies. 80 percent of firms have it on; 23 percent of comments come from it. Review is the chokepoint where AI is still "live but unusable".

What actually hurts is what the company chooses to measure

The paper does not argue that AI coding agents are weak. It argues that organisations are measuring the wrong thing. As long as the dashboard reads PR count, commit count, and lines of code, every coding agent becomes a beautiful performer on the wrong stage. Move the metrics to cost-per-resolved-issue, PR-to-merge time, and post-merge rollback rate, and the same agents suddenly look very different in different shops.

There is a deeper signal here. Agents change the shape of the bottleneck. The bottleneck used to be how fast one person can type; it is now how deep the review queue is. An organisation that does not reallocate human attention to the irreversible actions — merging, force-pushing, deleting branches, editing repo settings — and instead lets agents absorb review as well, will see that 49 percent extend further, not contract.

How seriously to take this

The piece, as relayed by Solidot, is a working paper, not peer-reviewed. Its empirical strength comes from the sample size: 700 firms, 300 million events is a degree of breadth no single-company A/B can match. Its weakness is selection bias: every company in the dataset runs Jellyfish tooling, which biases toward mid-market software firms that already buy engineering analytics.

Even with those caveats, this is the most cross-sectional and least ideological answer to "do AI coding tools actually work?" that has surfaced this year. The takeaway is clean: the next twelve to twenty-four months of differentiation will not come from whether a team uses AI coding agents. It will come from who redesigns the review queue and the metrics first.

Reference: