About a quarter of all pull requests reviewed in any given month were completely or largely AI-generated—up from fewer than 1% early last year. Detection used signals including branch names and PR descriptions.
Supporting evidence
Original excerpt
And I came to the result that about a quarter of all the poll requests that Graal was reviewing in any given month were completely or at least largely generated by AI. Then I backtracked this data to the last 12 months and it turns out that this number is growing really fast. In fact, early last year, fewer than 1% of pull requests had any evidence of being completely AI generated.
Revert rates show minimal difference between human and agent PRs
Tracking revert rates as a proxy for bad pull requests found one agent reverted about one per thousand, another 3.5 per thousand, and humans around 2.5 per thousand. There doesn't seem to be a very big difference between the rate at which pull requests were reverted from people versus agents in the study.
Supporting evidence
Original excerpt
Codex PR is reverted about one out of every thousand poll requests. Devon once every three and a half times every thousand poll requests. Humans right in the middle at about two and a half. So there doesn't seem to be very big difference between the rate at which poll requests were reverted from people versus agents in my study.
AI coding agents exhibit distinct qualitative failure patterns
There was significant variation in the types of failures agents produced compared to humans. One agent was one and a half times more likely to produce a SQL injection error than humans, while another was about half as likely to produce an auth bypass issue.
Supporting evidence
Original excerpt
Claude is one and a half times more likely to produce a SQL injection error than humans. Devon is about half as likely as humans to produce a off bypass issue. I found it very interesting that there was this much variation in how these agents were performing and how different their failure modes were from humans.
nafyi.