How to Evaluate AI Coding Agents: The Same-Task Method We Use
To evaluate AI coding agents, run the same small, verifiable task through every agent you are considering, record each run against a fixed schema, and let the records — not the demos — make the decision. That is the whole method; the craft is in the details: the task must be small enough to verify and large enough to force the agent to read your code, the prompt must not change between tools, the timing boundaries must be explicit, and incomplete runs must be excluded by rule rather than by mood. This page publishes the exact protocol we use for our own benchmark, including the mistakes it took to get it right.
Step 1–2: the task and the prompt
Our canonical task: in a small TypeScript API, add a validated /health endpoint that returns service status and a timestamp, plus a unit test that proves it responds, then run the tests and summarize the diff and assumptions. It is small enough that one person can verify the result in minutes, and it still forces the behaviors that separate an agent from an autocomplete: reading existing files, editing code, running a command, and reporting honestly. Whatever task you choose, freeze the wording. The moment you rephrase the prompt for one tool, you are benchmarking your prompts, not the agents.
Step 3: the record schema
| Field | What it tells you |
|---|---|
| toolVersion + checkedAt | Agents change weekly; an undated result is a rumor |
| usefulResultTimeMinutes | Time to a result you could actually use — not time to first output |
| Manual interventions + review corrections | The best predictors of what the agent will cost a team day to day |
| readRelevantFiles / producedReviewableDiff / ranTests | Behavior, not claims: did it act like an agent or guess like a chatbot |
| firstAttemptPassed + humanIntervention | Whether the loop closed unattended, and how much of you it needed |
| rawEvidencePath | No raw transcript and diff, no citation — this is the anti-fabrication field |
Step 4–5: timing boundaries and the publish gate
Two rules keep the numbers honest. First, declare what the clock includes. In our own 2026-07-10 run, one record measured only from the implementation patch through verification while the others included the full agent session — a timing boundary difference that makes those times incomparable, which is why our published table says so in the caption instead of hoping nobody notices. Environment friction is real (one tool needed device re-authorization before it could start; another had to be installed and authenticated from scratch) and it belongs outside the task clock. Second, gate by completeness: a record only counts if it is marked complete, carries a valid date, and points to raw evidence. Our validator deliberately refuses to turn partial work into a positive ranking — and a failed first attempt may still be published as evidence when the repair path and review burden are recorded.
Step 6: what our own run taught us
Our four-agent run on the canonical task ended with every agent passing on the first attempt with zero interventions. That is a good result and a bad benchmark: a task every agent passes cannot rank them. It still produced real information — all four closed the read-edit-test loop unattended, and one run yielded a concrete cost point of $0.28 in metered usage — but the discriminating power is gone, so the next iteration of our protocol escalates task difficulty until the agents separate. Expect the same on your side: calibrate the task until it is hard enough that at least one candidate fails, because the failures are where the evaluation lives. One more honest limit: single runs have no statistical power. They beat vendor demos and secondhand screenshots, but treat every number as one observation, not a distribution.
Frequently asked questions
How do I compare AI coding agents fairly?
Same task, same prompt word for word, same record schema, explicit timing boundaries, and a completeness gate on which runs count. Change any one of those between tools and the comparison quietly becomes about your setup instead of the agents.
What metrics matter most when evaluating a coding agent?
Useful-result time (not time to first output), manual interventions, and review corrections predict day-to-day cost better than raw speed. Behavioral checks — did it read the relevant files, produce a reviewable diff, run the tests — separate agents from chatbots.
How big should the evaluation task be?
Small enough that a human can verify the result in minutes, large enough to force reading existing code. Then calibrate upward: if every agent passes, the task has no discriminating power — escalate difficulty until at least one candidate fails, and read the failure.
Should I trust published AI coding benchmarks?
Trust the ones that publish their prompt, timing boundaries, per-run records, and raw evidence — and that admit what their task cannot show. Any benchmark that ranks agents without stating its timing boundary or sample size is marketing with a table in it, including when it flatters the tool you already like.
Bottom line: an agent evaluation is a small engineering project, not a vibe check. One task, one frozen prompt, one schema, one gate — then let the records argue. If you adopt nothing else from this page, adopt the two rules that cost us the most to learn: declare your timing boundaries, and escalate the task until somebody fails.