AI Coding Benchmark and Risk Library
Direct answer: use each finding only inside its stated sample and limitation. Availability checks never create winner claims, and pending app-builder runs remain non-results.
| Category | Tools | Metric | Evidence class | Finding | Limitation |
|---|---|---|---|---|---|
| coding-agent | Claude Code, Codex, Cursor, GitHub Copilot | task-result | reproducible-experiment | All four retained runs identified the root-cause module and produced a first-attempt patch that passed independent verification; Cursor did not run tests inside the restricted non-interactive mode. | One small fixture and one run per surface cannot establish general coding quality, speed, or reliability; Cursor's shell restriction is a mode boundary rather than a product-wide verdict. |
| code-review | Claude Code, GitHub Copilot | false-positive | reproducible-experiment | Claude Code mapped 8 of 9 seeded defects and one false-positive trap; GitHub Copilot CLI mapped 5 of 9 seeded defects and one false-positive trap. | A single synthetic diff and one run per tool do not establish general review recall or false-positive rates; the CLI surfaces are not the same as hosted pull-request bots. |
| app-builder | v0, Lovable, Bolt.new, Replit Ai | availability | reproducible-experiment | All four records remain pending, so no build, revision, export, deployment, handoff, or performance result is publishable. | This is an evidence-availability check, not evidence that any product is unavailable or failed the task. |
| coding-agent | Windsurf, Devin | availability | official-source | Windsurf pricing and getting-started URLs redirect to Devin pricing and Devin Desktop documentation. | A redirect establishes current public packaging, not the contract terms or migration status of every existing customer. |
| coding-agent | Cursor | security | official-source | Cursor documents a Privacy Mode that prevents training on customer code and can be enforced by teams. | This records a documented control, not an independent audit of implementation or a comparison with other products. |
Methodology and evidence
Limitations
Local experiments are single runs on small fixtures, not distributions. Vendor pages are not independent audits. Model-only benchmarks are not mapped onto consumer product surfaces.
Sources and evidence
Sources
- Cognition sourceChecked 2026-07-13High volatility
Windsurf pricing and getting-started URLs redirect to Devin pricing and Devin Desktop documentation.
- Cursor sourceChecked 2026-07-13High volatility
Cursor documents a Privacy Mode that prevents training on customer code and can be enforced by teams.
Evidence
- BenchmarkChecked 2026-07-11
One small fixture and one run per surface cannot establish general coding quality, speed, or reliability; Cursor's shell restriction is a mode boundary rather than a product-wide verdict.
Methodology - BenchmarkChecked 2026-07-11
A single synthetic diff and one run per tool do not establish general review recall or false-positive rates; the CLI surfaces are not the same as hosted pull-request bots.
Methodology - BenchmarkChecked 2026-07-11
This is an evidence-availability check, not evidence that any product is unavailable or failed the task.
Methodology - Decision matrixChecked 2026-07-13
A redirect establishes current public packaging, not the contract terms or migration status of every existing customer.
Methodology - Decision matrixChecked 2026-07-13
This records a documented control, not an independent audit of implementation or a comparison with other products.
Methodology