How to Review Code with AI (2026): A PR Workflow Tested Against Nine Seeded Defects
How to review code with AI, in one honest sentence: use the AI as a first-pass filter on the diff, verify every finding against the code before acting on it, and keep human review for the semantics AI reliably misses. That framing is not caution for its own sake — it is what our own test forced us to write. On 2026-07-11 we built a pull request with nine seeded defects (SQL injection, a hardcoded live key, a missing await, an off-by-one, swallowed errors, and four more) plus two decoys, and asked four CLI coding agents to review it with the same prompt. Three found eight of nine with zero fabricated findings; one found four of nine and confidently reported two defects that were not in the code; all four missed the same subtle timezone bug. AI review is genuinely useful — and it is not a merge gate.
How we tested (and why you can check us)
The fixture is a small TypeScript users service; the PR narrative is realistic — it adds user search and CSV export. Into that diff we planted nine seeded defects spanning security (string-interpolated SQL, a hardcoded live-style API key), correctness (a fire-and-forget async notification, an off-by-one in pagination, unescaped CSV), error handling (a swallowed catch, a leaked file handle), privacy (PII logged on every search), and one deliberately subtle case: a date filter that mixes UTC parsing with local-time parsing. We added two decoys — intentional code that looks suspicious but is documented as deliberate — to measure false positives, and we scored four CLI reviewers (Claude Code 2.1.117, codex-cli 0.144.1, cursor-agent 2026.07.09, GitHub Copilot CLI 1.0.70) in read-only mode against a ground-truth ledger written before any tool ran. The ledger, the prompt, and all four raw outputs are published in our repository; every claim in this article traces to them.
Step 1: give the reviewer the diff and the repository
The single highest-leverage setup decision is context. A diff alone hides the conventions the change violates: in our fixture, the seeded pagination bug is only obviously wrong because the same file's existing function computes offsets differently — a reviewer that can read the whole file catches the inconsistency, a reviewer that sees ten diff lines cannot. Point the tool at the branch, not a pasted fragment, and say what the PR claims to do (ours: the PR adds user search and CSV export), because several defects are only defects relative to that intent. All four tools in our test read beyond the diff when given the chance, and the three strongest used the surrounding code to justify findings.
Step 2: triage — findings are candidates, not verdicts
Before you fix anything an AI reviewer reports, open the file and confirm the claim is true. That rule exists because of what we observed: one reviewer in our test reported that a template string was unclosed and the code could not parse — the file was perfectly valid — and separately claimed a query would fail at runtime for missing parameters, when the real problem was the opposite: the query ran fine and was injectable. Both findings sounded specific and confident. A hallucinated finding costs you twice — once to chase it, and once more in the trust you stop extending to the tool's real findings. Triage order that works: verify high-severity claims first (they are also where AI is most accurate — all four tools caught all three of our high-severity seeds), batch the mechanical fixes, and send anything the tool says about behavior to a test before believing it.
Step 3: know the shared blind spot
The most important negative result in our test: the timezone defect was found by zero of four tools. The seeded bug mixes `new Date` parsing of a date-only string (interpreted as UTC) with parsing of a timestamp string (interpreted as local time), so records near the boundary are silently misclassified. Nothing about it is exotic — it is the kind of bug that ships and surfaces weeks later as a support ticket. It evaded every reviewer because spotting it requires holding two parser behaviors in mind at once, across a boundary the diff does not mark. Treat this class — subtle semantics that span an implicit boundary — as human territory, and let it shape what your human reviewers spend their attention on now that AI handles the mechanical pass.
| Review scenario | What to do | Why (from our test) |
|---|---|---|
| Pre-PR self-review | Run the AI on your branch before requesting review | Catches the mechanical majority cheaply — 8/9 for the strongest tools |
| Security-sensitive diff | AI first pass, then human security review regardless of a clean report | High-severity hits were AI's best category — but 4/9 coverage exists in the wild |
| Date, money, or boundary logic | Assume the AI missed it; write a boundary test yourself | The timezone seed went 0-for-4 |
| A finding you cannot reproduce | Demand a failing case before fixing; it may be hallucinated | Two confident findings in our run described code that did not exist |
Where this workflow stops working
Three failure boundaries, stated plainly. First, do not merge on a clean AI review alone: the weakest coverage we measured was 4/9, which means a green report can coexist with five live defects, including everything in the privacy and error-handling categories that tool missed. Second, do not run AI review as the only reviewer on code whose failure modes are semantic — pricing rules, date arithmetic, permission checks — because the one defect class every tool missed lives exactly there. Third, do not treat this article as a ranking: we tested CLI agents acting as reviewers, one run each, on one seeded fixture. SaaS review products that live inside the PR flow are a different surface we have not measured, and a single run has no statistical power. Our numbers are dated, sourced, and reproducible — and they are still one observation each.
Frequently asked questions
Can AI review code reliably?
For mechanical defects, yes with verification: in our seeded test the three strongest CLI reviewers found 8 of 9 planted defects with zero fabricated findings, and all four tools caught all three high-severity seeds. For subtle semantic defects, no: the timezone bug we planted was missed by every tool. Use AI as the first pass, not the only pass.
What do AI code reviewers miss?
In our test, the shared miss was cross-boundary semantics: a date filter mixing UTC and local-time parsing, missed by all four tools. Weaker coverage also skipped PII logging, swallowed errors, resource leaks, and CSV escaping. What they caught best was mechanical and high-severity: injection, a hardcoded credential, a missing await.
Do AI code reviewers make up bugs?
One tool in our run did — twice: it reported a syntax error in a perfectly valid file and a runtime failure that could not happen. The three stronger tools fabricated nothing. So hallucinated findings are real but tool-dependent; the defense is the same either way — confirm every finding against the code before acting.
Should AI review replace human code review?
No — it should change what humans review. Let the AI do the mechanical sweep (it is fast and strongest on high-severity defects), and redirect saved human attention to intent, architecture, and boundary semantics — the layer where our test measured a 0-for-4 AI detection rate.
Bottom line: AI code review earns a permanent place in the PR workflow as a first-pass filter — it caught the majority of what we planted, fast and cheap. It has not earned the right to merge unverified, because the same test that proved its value also produced two confident hallucinations, a 4/9 coverage floor, and one subtle bug that beat every tool. Run it first, verify everything, and spend your human reviewers where the machines went 0-for-4.