AI Code Review Accuracy (2026): We Seeded 9 Defects — What Four Reviewers Found, Missed, and Made Up
How accurate is AI code review? On 2026-07-11 we measured it the only way that produces a checkable number: we seeded a realistic pull request with nine known defects and two decoys, wrote the ground-truth ledger before any tool ran, and gave four CLI reviewers the identical prompt in read-only mode. The result splits the accuracy question into three different risks. Detection: Claude Code, Codex, and Cursor each found 8/9; GitHub Copilot CLI found 4/9. Fabrication: the three stronger tools invented nothing; Copilot CLI reported two defects that did not exist in the code. And one shared blind spot: the subtlest seed — a timezone-mixing date filter — was missed by all four. False positives of the pedantic kind, the ones everyone fears, barely appeared at all.
Method: ground truth first, tools second
The fixture is a TypeScript users service; the PR adds user search and CSV export. The nine seeds span security, correctness, error handling, privacy, and one deliberately subtle semantic case. The two decoys are code that looks wrong but is documented as intentional — a sequential await loop required for stable pagination, and a defensive integer coercion — planted specifically to measure false positives. Scoring rules were fixed in the ledger before the runs: a finding counts only with correct location and correct causal story; flagging a decoy counts as a false positive unless the tool makes a substantive correctness argument rather than a style complaint; anything a tool reports beyond the seeds gets human re-verification and, if real, is recorded as an extra rather than retroactively changing the 9-count. The reviewers: Claude Code 2.1.117, codex-cli 0.144.1, cursor-agent 2026.07.09, GitHub Copilot CLI 1.0.70 — one run each, same prompt, read-only.
The results, per tool
| Reviewer (version) | Seeded defects found | High-severity (of 3) | Hallucinated findings | Verified extra true defects | Review time |
|---|---|---|---|---|---|
| Claude Code 2.1.117 | 8/9 | 3/3 | 0 | 2 | 71s |
| codex-cli 0.144.1 | 8/9 | 3/3 | 0 | 6 | 595s |
| cursor-agent 2026.07.09 | 8/9 | 3/3 | 0 | 3 | 43s |
| GitHub Copilot CLI 1.0.70 | 4/9 | 3/3 | 2 | 0 | 41s |
The two hallucinations, in detail
Both fabricated findings came from the same run and both sounded rigorous. The first claimed the search query would fail at runtime because no parameters array was passed — in fact the query runs fine, because the values are interpolated into the SQL string, which is precisely the injection defect the same tool had already (correctly) reported one finding earlier. Wrong causal story, duplicated location, stated as fact. The second claimed a template string was unclosed and the file could not parse; the file is valid TypeScript, and every other tool read it without comment — the reviewer appears to have misread the diff rendering. This is the signature of the hallucinated-finding failure mode: specific line numbers, confident severity labels, plausible mechanics, nonexistent defect. It is more dangerous than a vague warning, because it survives a skim. The defense is procedural, not technological: no finding gets acted on until someone reproduces it against the code.
The defect missed by all four
The date-filter seed mixes two parser behaviors: a date-only string is parsed as UTC midnight while the records' timestamp strings are parsed as local time, so rows near the boundary are silently misclassified. No tool flagged it — not the two that spent minutes reading the repository, not the two that answered in under a minute. The pattern generalizes: every seed that all four tools caught can be seen in the changed lines themselves; the one seed that none caught requires knowing an API's implicit behavior across a boundary the diff never marks. If you maintain code where dates, money, or permissions cross boundaries, that class of defect is still yours. Two honest notes on the other side of the ledger: our decoys fooled nobody (the flags they did draw were substantive correctness arguments, which we did not score as false positives), and three of the four tools independently found a real pagination-semantics bug we had not intentionally planted — verified by us and recorded as a tenth, discovered defect. AI review can find true problems its authors did not know about, and still miss the one a senior reviewer would circle in red.
Scope and limits
What this data is: a controlled, ground-truth-first measurement of four CLI coding agents acting as PR reviewers, with every raw output published. What it is not: a ranking of code review products. We did not test SaaS review bots that live inside the pull-request flow; a tool's CLI persona and its PR-integrated product can behave differently; and every number here is one run — rerunning could move any cell. We publish the fixture design, prompt, and ledger precisely so the numbers can be challenged, and we retire this fixture from future rounds now that it is public.
Frequently asked questions
How accurate is AI code review?
In our 2026-07-11 seeded test: 8/9 detection for the three strongest CLI reviewers, 4/9 for the weakest, 3/3 on high-severity defects for all four, and 0/4 on the subtlest semantic seed. Accuracy is not one number — detection, fabrication, and blind spots move independently.
What is a hallucinated finding in AI code review?
A confidently reported defect that does not exist — in our run, a claimed syntax error in a valid file and a claimed runtime failure that could not happen. It differs from a pedantic false positive: it is not an overreaction to real code, it is a description of imaginary code. Reproduce before you fix.
What kinds of bugs do AI reviewers catch best?
Mechanical defects visible in the changed lines: SQL injection, hardcoded credentials, missing await, inconsistent pagination math, swallowed errors. All three of our high-severity seeds were caught by all four tools. The miss pattern was the inverse: semantics that span an implicit boundary.
How do I reduce false positives from AI code review?
Document intent in the code: our two decoys carried comments explaining why the suspicious-looking pattern was deliberate, and no tool scored a style complaint against them. Then enforce reproduce-before-fix for every finding — it filters hallucinations at triage instead of in your diff.
Bottom line: the accuracy story is better than the skeptics say and worse than the demos imply. On one seeded PR, the strong tools caught nearly everything mechanical, invented nothing, and even surfaced a real bug we had not planted — while the weakest run missed more than half the seeds and fabricated two findings, and every tool walked past the same semantic defect. Choose your reviewer with data like this, then run it inside a workflow that assumes both failure modes exist.