By Aijaz Alam
Does AI make developers faster?
It depends on the study. In METR's randomized trial of 16 experienced open-source developers (February to June 2025), tasks took 19% longer when AI tools were allowed. A 2022 GitHub Copilot experiment with 95 freelancers finished one task 55.8% faster, and a 2024 Google trial of 96 engineers about 21% faster.
- 19%Longer on tasks for experienced developers allowed AI toolsMETR trial, Feb-Jun 2025; METR now calls it out of date
- 55.8%Faster on one JavaScript task with GitHub CopilotFreelancer experiment, 2022, published 2023
- 21%Less time on an enterprise task for Google engineers with AI (about)Google trial, 96 engineers, published Oct 2024
- -18%METR's 2026 speedup estimate for returning developers, in METR's termsConfidence interval -38% to +9%; METR calls it weak evidence
- 56%Average security pass rate of AI-written codeVeracode tests of 100+ models since 2023 (vendor)
The numbers
The full breakdown
Controlled studies of AI and developer speed
| Study | When | Who took part | Result with AI |
|---|---|---|---|
| Peng and others, GitHub Copilot experiment (arXiv 2302.06590) | May to Jun 2022 | 95 freelance developers recruited on Upwork, one JavaScript server task | 55.8% faster (71 vs 161 minutes); 95% CI 21% to 89% |
| Paradis and others, Google randomized trial (arXiv 2410.12944) | Published Oct 2024 | 96 full-time Google software engineers, one complex enterprise task | About 21% less time on task; wide confidence interval |
| METR randomized trial | Feb to Jun 2025 | 16 experienced open-source developers, 246 real issues in their own repositories | 19% longer; CI +2% to +39% |
| METR follow-up, returning developers | From Aug 2025 | 10 developers from the first study, late-2025 tools | Estimated speedup of -18% in METR's terms (CI -38% to +9%); METR reads its raw results as some evidence for speedup |
| METR follow-up, new developers | From Aug 2025 | 47 newly recruited developers, late-2025 tools | Estimated speedup of -4% in METR's terms (CI -15% to +9%) |
Each study randomly assigned tasks or people to AI or no AI and measured time, not self-reports. A confidence interval that crosses zero means the study cannot rule out no effect. METR marks its 2025 result as out of date and says the 2026 data is only very weak evidence because some developers withheld tasks they did not want to do without AI.
METR 2025 trial: what developers expected and what was measured
| Measure | Change in speed with AI |
|---|---|
| Developers' forecast before the study | 24% faster |
| Developers' own estimate after the study | 20% faster |
| Measured in the trial | 19% slower (tasks took 19% longer) |
16 experienced open-source developers, February to June 2025.
Security of AI-written code
| Measure | Value | Period |
|---|---|---|
| Average security pass rate of code written by AI models | 56% | 2023 to 2026 |
Veracode, 2026 GenAI Code Security Report: more than 100 models tested on Veracode's own task set. A vendor benchmark of models, not a trial with developers; Veracode sells code security tools.
Why sources differ
Same question, different answers
| Number | When | Who measured | Why it differs |
|---|---|---|---|
| 55.8% faster | 2022 test, published 2023 | GitHub Copilot experiment (arXiv 2302.06590) | 95 Upwork freelancers, one new JavaScript task, early Copilot. |
| About 21% faster | Published Oct 2024 | Google randomized trial (arXiv 2410.12944) | 96 Google engineers on one enterprise-grade task, three AI features. |
| 19% slower | Feb to Jun 2025 | METR randomized trial | 16 experienced developers on their own large repositories, early-2025 tools; METR now calls it out of date. |
| Speedup of -18% (METR's terms), returning developers | Aug 2025 to early 2026 | METR update, Feb 2026 | Late-2025 tools; METR reads it as some evidence of speed-up but says selection effects make it weak evidence. |
| 73% daily use | Jun to Aug 2026 | Stack Overflow Developer Survey 2026 | Self-reported use, not measured speed. |
The studies disagree because they test different people on different work with different tools. The 2022 Copilot experiment gave freelancers one fresh, self-contained task, and it used an early version of the tool. Google's trial also used a single complex, enterprise-grade task and tested three AI features with Google's own engineers. METR did the opposite: it watched experienced developers fix real issues in large codebases they already knew well, and found them 19% slower with early-2025 tools. METR's follow-up with late-2025 tools points the other way for the same returning developers, but its confidence interval runs from 38% faster to 9% slower, and 30% to 50% of developers said they held back some tasks they did not want to do without AI, so METR treats its estimate as likely too low. Survey numbers such as Stack Overflow's 73% daily use measure adoption, not speed, and developers are poor judges of their own speed: METR's developers believed AI made them 20% faster when it made them slower.
What it means
How to use these numbers
There is no single answer yet. Controlled studies have found AI coding tools making developers anywhere from 19% slower to 55.8% faster, and the size of the effect depends on the task, the developer's experience with the codebase and the tool version.
The pattern so far: the speed-ups come from single, well-defined tasks set by the researchers, and the slowdown from experienced developers working on real issues in large codebases they already knew. With so few studies, that is a pattern to watch, not a rule. Newer tools may have shifted this; METR's 2026 data leans toward a speed-up but is weak evidence.
For a small business paying for software work, do not assume AI makes a project cheaper by a fixed share. Ask your developer or agency how they use it and how they check its output: in Veracode's vendor tests, AI-written code passed security checks only 56% of the time. For managers, do not rely on how fast developers feel; in METR's trial they believed AI had sped them up by 20% when the measured result was 19% slower.
Method and sources
How it was measured
Method
- Studies
- GitHub Copilot experiment (Peng, Kalliamvakou, Cihon, Demirer; arXiv 2302.06590); Google randomized trial (Paradis and others; arXiv 2410.12944); METR randomized trial (July 2025) and its update (24 Feb 2026)
- Design
- Randomized controlled trials: tasks or people randomly given AI access or not, with time to finish measured, not self-reported
- Samples
- 95 freelancers (2022); 96 Google engineers (2024); 16 experienced open-source developers and 246 issues (2025); 57 developers, 143 repositories and 800+ tasks in METR's follow-up
- Code quality
- Veracode 2026 GenAI Code Security Report: vendor benchmark of more than 100 AI models since 2023
- Checked at source
- 31 October 2026, on metr.org and arxiv.org; Veracode and the Copilot paper on 23 October 2026
- Next release
- No schedule; METR says it is changing its study design and will publish new results
- Last changed
- 30 Oct 2026
- Data files
- CSV · JSON (stable links)
Sources
- METR research notes, METR (Model Evaluation and Threat Research) Study
- arXiv preprint server, Cornell University (arXiv) Study
- GenAI Code Security Report, Veracode Vendor data
- Stack Overflow Developer Survey, Stack Overflow Survey
Cite this page
APA
MLA
HTML link
FAQ
More questions people ask
Why did METR find developers slower with AI?
Its 16 developers were experienced maintainers working on large open-source projects they knew well, using early-2025 tools on real issues in code they knew well. That is a very different setting from the one-task experiments that found speed-ups. METR now says these results are out of date.
Did METR reverse its finding?
Not exactly. Its February 2026 update estimates an 18% shorter task time for returning developers with late-2025 tools, but with a confidence interval from 38% faster to 9% slower. METR says developers are likely more sped up now, but that its data is only very weak evidence of how much.
Is the 55.8% Copilot figure a measure of overall productivity?
No. It is the time difference on one JavaScript task (71 minutes with Copilot against 161 without) for freelancers recruited on Upwork, with a 95% confidence interval of 21% to 89%.
Is AI-written code secure?
Often not. Veracode's tests of more than 100 AI models since 2023 give an average security pass rate of 56%, so about 4 in 10 coding tasks produce code with a security flaw. It is a vendor benchmark, not a study of developers.
Every number on this page was checked at its original source (METR (Model Evaluation and Threat Research), Cornell University (arXiv), Veracode, Stack Overflow). Free to cite with a link to this page.