Fully Autonomous Pull Requests Didn't Lower the Bar

Last month, more than a third of the pull requests merged through our platform had no human reviewer. Our agent was the only thing that read them. About 8% of them were merged fully autonomously by us after we approved them.

We can measure this because we ship the auto-approve and auto-merge paths, which gives us unique data nobody else has. About 6k pull requests a week merged with our agent as the only reviewer. They merged in a median of 0.5 hours against roughly 2.5 hours for human-reviewed ones. They were also larger, averaging about 2k changed lines against 1.5k, so this is not teams skipping review on typo fixes.

The review did not get worse when the agent was the only one looking. About 13% of pull requests had changes requested in both groups. The agent blocked slightly more often with no human present, 2% against 1.7%. Developers dismissed its findings less often too, 10% against 16%.