A year ago, METR produced one of the most cited findings in the debate over AI coding tools: in a randomized controlled trial of 16 experienced open-source developers working on mature codebases, letting people use AI assistants made them 19% slower, not faster, even though those same developers had predicted beforehand that AI would cut their completion time by 24%, according to METR’s July 2025 study. Outside economists and ML experts, extrapolating from hype rather than task-level data, had guessed AI would save closer to 38–39% of the time. The gap between expectation and outcome was itself the finding. Afterward, the developers still believed AI had saved them 20% of their time, a belief the stopwatch flatly contradicted.
| Category | Change in completion time (%, positive = time saved) |
|---|---|
| Developers' prediction (before) | 24 |
| Actual RCT result | -19 |
| Developers' belief (after) | 20 |
Naturally, METR wanted to know if the result would hold up with a larger sample and more time for developers to adapt to newer tools. It launched a follow-up in August 2025. It has not been able to finish it. In a post explaining why it is changing the study’s design, METR reports that the experiment produced an unreliable signal because too many developers simply refused to be randomized into the no-AI condition. The researchers describe “a significant increase in developers choosing not to participate in the study because they do not wish to work without AI,” and their surveys found that 30 to 50 percent of participants skipped submitting some tasks rather than complete them unassisted. One developer’s explanation, quoted by the team, captures the mood better than any statistic could:
“I’d like to help provide updated data on this question but also I really like using AI!”
When the counterfactual disappears
What makes this more than an amusing footnote is what it implies about the object being studied. A randomized controlled trial only works if researchers can actually construct the condition they want to compare against — in this case, a developer doing real work with no AI assistance at all. If a large enough share of a professional population won’t accept being placed in that condition, the counterfactual itself becomes empirically inaccessible, not just personally unpleasant. That is a different and arguably more significant problem than any productivity percentage, because it suggests AI use has moved from being a tool developers evaluate against alternatives to being a baseline behavior they will actively resist having removed, even temporarily and even in the name of research they claim to support.
TechCrunch’s coverage frames this bluntly: developer reliance on AI coding tools has outrun the evidence for how much those tools actually help, and the dependency carries its own risk — atrophy of unassisted debugging skills, accumulated technical debt from code nobody fully reasoned through — precisely at the moment it becomes hardest to study honestly. It is a warning shaped like a paradox: the more entrenched the habit, the less anyone can measure what the habit is costing.
A field that cannot construct a control group is not a field with a settled answer.
Self-reported gains, and the one group that didn’t believe them
The behavioral entrenchment sits awkwardly next to what developers say about their own output. A companion survey of 349 technical workers, run around the same time, found a median self-reported gain of a 2x increase in the value of their work from using AI — a number that, if taken at face value, would make the original 19%-slower finding look like an anomaly rather than a warning. But METR’s own staff, arguably the people best positioned to interpret the lab’s controlled-trial data because they work with it daily, reported the lowest self-rated productivity gains of any group surveyed, a detail METR itself flagged as a reason to treat the self-reported 2x figure with skepticism rather than as corroboration.
That asymmetry is the real substance of the story. It is not that AI coding tools obviously help or obviously don’t; it is that the population best equipped to answer that question, professional developers, has become too attached to the tools to let researchers isolate their effect, while the group closest to the underlying evidence is the least impressed by its own optimism. A field that cannot construct a control group is not a field with a settled answer. It is a field where the belief has outpaced the measurement, and where the emptiest chair in the office — the one nobody wants to sit in, unassisted, stopwatch running — has become the hardest data point to collect.
