AI Code Review: What to Automate and What Stays Human
AI code review is an automated reviewer that reads your diff and comments on it before a human does. It is good at the mechanical pass: unhandled error paths, missing tests, secrets in source, duplicated logic that an existing helper already covers, drift from your own conventions. It is bad at judging whether the change was worth making. Run it as the first lane, never as the last word.
TL;DR
- Review became the bottleneck because generation got cheap. GitClear's analysis of 211 million changed lines found duplicated code blocks rose eightfold in 2024, while moved (refactored) code fell from 24.8 percent of changed lines in 2021 to 9.5 percent in 2024.
- Speed is not automatic. METR's randomized trial put 16 experienced developers through 246 real tasks and measured them 19 percent slower with AI tools, while they believed they were 20 percent faster.
- The 2025 DORA report has 90 percent of respondents using AI at work, with adoption now linked to higher throughput and still linked to lower delivery stability. More change, same fragility.
- Precision beats coverage. Google's ML review suggestions target about 50 percent precision, and about 7.5 percent of all reviewer comments get resolved by an applied suggestion.
- Small diffs are the cheat code. The SmartBear study at Cisco (2,500 reviews, 3.2 million lines) puts the working range at under 400 lines and under 500 lines per hour for 70 to 90 percent defect discovery.
The three places review actually happens
Most teams turn on the second one and stop there, which leaves the cheapest win on the table.
In the editor. Suggestions as you type. Useful, invisible, not a review. It never sees the whole change.
On the pull request. The bot that posts comments on the diff. This is what people mean by AI code review, and it is the lane that generates both the value and the noise.
Before the pull request exists. A self review pass you run on your agent's diff, in a fresh session, before you open anything. This is the highest return step and almost nobody does it, because the agent that just wrote the code feels finished. Ten minutes here removes the comments a human would otherwise have to write.
Why the review layer got load-bearing
The cost of producing code dropped. The cost of verifying it did not.
GitClear's read of 211 million changed lines is the clearest signal. 2024 was the first year on record where within-commit copy and paste exceeded moved code, meaning teams are adding more duplicate blocks than they are reorganizing existing ones. Assistants are very good at inserting a working block and much worse at noticing that the function already exists forty files away, partly because it never fit in the context window.
DORA's 2025 data says the same thing from the delivery side. AI adoption finally correlates with higher throughput, and it still correlates with lower stability. You ship more, and more of it comes back. Review is one of the few places you can catch that before a customer does.
Split the review into two lanes
The single decision that makes AI code review work is deciding what the bot is allowed to have an opinion about.
The machine lane is everything mechanical, repeated, and checkable against a rule: error paths, null handling, an unbounded query, a new route with no auth check, a missing test for a new branch, a secret in source, logic that duplicates an existing helper. These are exactly the comments senior reviewers get tired of writing.
The human lane is judgment: is this the right change, what is the blast radius, is this an API we can live with in a year, does the test assert the thing that matters or just the thing that is easy. No bot votes here.
Write that split down somewhere the tool reads. A reviewer with no boundary will comment on import order forever, and the day it finds a real authentication hole, nobody will be reading.
Make the diff reviewable before anything reviews it
The best review tooling in the world loses to a 2,000 line pull request.
The Cisco numbers are old and still true, because they measure people, not tools: 200 to 400 lines per review, under 500 lines per hour, 60 to 90 minutes before detection falls off. What changed is how fast you can now exceed that. An agent can produce a week of diff in an afternoon.
Three constraints that keep changes inside the window:
- Put a size rule in your agent instructions. Something like "stop and open a pull request at roughly 300 changed lines, then wait" belongs in
CLAUDE.mdorAGENTS.md, not in your head. - Separate refactors from behavior. A commit that moves code and a commit that changes what it does are two different reviews. Mixed together, the behavior change hides in the diff.
- Make the agent write the PR description with the reasoning and the tradeoffs it rejected. The reviewer's first question is always why, and the model is the only one who can still answer it cheaply.
Write the rules down or the bot invents its own
A generic reviewer gives generic comments. What turns it into your reviewer is repository facts it cannot infer from a diff. Keep them in one short file the tool loads on every run.
# review-rules.md
COMMENT ON: correctness, security, data loss, missing test for changed behavior.
NEVER COMMENT ON: formatting (prettier owns it), naming taste, import order.
BLOCKER: unbounded query, new route without an auth check, secret in source,
migration with no rollback, money handled as a float.
NOTE: duplicated logic an existing helper covers, swallowed error, dead branch.
REPO FACTS
- Money is integer cents everywhere. Never a float.
- All timestamps are UTC. The UI converts, the API never does.
- Never call the billing API from a request handler. Queue it.
- `db.raw()` is banned outside `src/db/migrations`.
Ten lines of repository facts outperform any amount of prompt tuning. Every one of them is a bug that already happened to you once.
Tune for precision, not recall
The failure mode of AI code review is not missing bugs. It is 40 comments per pull request, of which four matter, and a team that stops reading by week three.
Google's system is the useful benchmark here precisely because it is conservative. Target precision around 50 percent, suggestions surfaced only above a confidence threshold, and the payoff is roughly 7.5 percent of review comments resolved by an applied edit. That is a well-tuned production system, not a demo, and it still only takes a slice.
So measure one number: the acted-on rate, meaning the share of bot comments that produced a code change or an explicit "no, because". Then run a weekly triage where every dismissed comment turns into one of two things: a new rule, or a suppression. If the acted-on rate sits under about 30 percent, turn categories off until it climbs.
That loop is the whole system. If you want the general pattern behind it, I wrote it up in WRITE LOOPS NOT PROMPTS, which is the same idea applied to any repeated agent job.
The gates that stay human
Some changes should never merge on a bot's approval, no matter how good the acted-on rate gets:
- Authentication, authorization, and permission checks.
- Database migrations, destructive queries, and anything that deletes.
- Money movement, pricing, and billing logic.
- Secrets, keys, and anything that changes what the deploy can reach.
- Public API contracts and anything another team builds against.
- Tool definitions and prompts for agents that act on untrusted input.
One more independence rule. Do not let the same session that wrote the code sign off on it. A model reviewing its own diff carries its assumptions forward and confirms the plan instead of testing it. Run the review in a clean context whose only inputs are the diff and the rules file, and for high-stakes changes, use a different model than the one that wrote it.
A setup you can ship this week
- Turn one PR reviewer on, in one repository, in comment-only mode. No blocking on day one.
- Write
review-rules.mdwith ten repository facts and an explicit never-comment-on list. - Add the pre-PR self review to your agent loop, in a fresh session, before the PR is opened.
- Triage for two weeks. Track the acted-on rate. Every dismissal becomes a rule or a suppression.
- Only then make the BLOCKER category blocking, and keep the human gates above exactly where they are.
Ship the first two steps in an afternoon. The value shows up in week three, when the human reviews start arriving with the boring problems already fixed.
The bottom line
AI code review is not a replacement reviewer. It is a way to move mechanical findings earlier and cheaper so the human attention lands on judgment, which is the part that was always scarce. Keep the diffs under 400 lines, write your repository facts down, tune for precision until people read the comments, and never let the model that wrote the change be the one that clears it.
If you want the next one of these when it lands, get it in the newsletter.
What is AI code review?
AI code review is an automated reviewer that reads a diff and leaves comments on it before a human does. It ships in three shapes. Inline in the editor while you type. As a bot on the pull request, which is what most people mean by the term. And as a self review pass you run on your own agent's diff before you open the PR at all. All three read the change plus some amount of repository context, then flag correctness problems, missing tests, unsafe input handling, and drift from your conventions. What none of them do is decide whether the change was worth making, which is why the useful setup is two lanes rather than one replacement.
Is AI code review accurate enough to trust?
It is accurate enough to be a first pass and not accurate enough to be the merge decision. Google's production system for resolving code review comments with machine learning runs at a target precision of about 50 percent, and even with that bar and enormous scale, roughly 7.5 percent of all reviewer comments end up resolved by an applied ML suggestion. That is a real win at Google's volume and it is also a clear statement about ceilings. Treat every finding as a claim to verify, tune the tool for precision instead of coverage, and keep a human gate on anything involving auth, money, migrations, or secrets.
Does AI code review actually save time?
Only when the diff is small and the rules are written down. METR ran a randomized trial with 16 experienced open source developers across 246 real tasks and found they were 19 percent slower with AI tools while believing they had been 20 percent faster. The lesson carries over to review: perceived speed is not measured speed. The measurable savings come from cutting review latency on mechanical findings, so the human reviewer opens a PR that already has its obvious problems fixed. Track the share of bot comments that produce a code change or an explicit decision. If that number sits under about 30 percent, the tool is costing you time, not saving it.
How big should a pull request be for review to work?
Under 400 changed lines, reviewed at under 500 lines per hour. Those numbers come from the SmartBear study of code review at Cisco, which covered about 2,500 reviews across 3.2 million lines of code and found that a 200 to 400 line review over 60 to 90 minutes yields 70 to 90 percent defect discovery, with detection falling off past those limits. AI does not change the human side of that math, it just makes it easier to blow past the limit in an afternoon. Put the constraint in your agent instructions: stop and open a PR at roughly 300 changed lines, and keep refactors in separate commits from behavior changes.
Should the same model that wrote the code also review it?
Not in the same session. A model reviewing a diff it just wrote carries its own assumptions forward, so it tends to confirm the plan rather than test it. Two cheap fixes. Run the review in a fresh context with no memory of writing the code, so the only input is the diff plus your rules. And where it matters, use a different model than the one that generated the change, since blind spots correlate within a model family more than across it. Independence is the whole value of a review, and it is the first thing an agent loop quietly removes.
OpusJake is Jake Schincariol's operating system for building with AI: agents, workflows, prompts, and the free resources behind them. Get the next move every week.