Why every AI code reviewer needs a failing repro
AI code review produces convincing noise. One rule fixes it: no finding counts without a failing test that reproduces it, plus a control that proves the test is valid.
Ask an AI model to review a pull request and you'll get findings. You'll always get findings. Some will be real bugs. Many will be confident, well-written concerns about problems that don't exist.
That's the core problem with AI code review: it's cheap to raise a concern and expensive to check it. If every finding needs a human to investigate, the reviewer isn't saving time, it's creating work.
The rule
On my projects, a review finding only counts if it comes with a failing test that reproduces it.
Not "this might cause a race condition". A test that triggers the race condition and fails. Not "this validator looks too loose". A test that sends the bad input and shows it getting through.
If the reviewer can't write the repro, the finding is downgraded to a question, not a blocker.
Why a control matters
A failing test proves something failed. It doesn't prove the code is wrong. Sometimes the test is wrong: the fixture is invalid in some second way, the mock is set up incorrectly, or the test is asserting the wrong thing.
So every repro needs a control: the same fixture, valid in every respect except the one field being attacked, and it should pass. If the control fails too, the problem is the test, not the code.
Declining false positives with evidence
The rule cuts both ways. When an implementer disagrees with a finding, "I disagree" isn't an answer either. They decline it with evidence. The strongest evidence is a mutation that fails nothing: break the code in exactly the way the finding describes, run the tests, and show that the claimed bug can't happen.
Every finding ends up in one of three states:
- Real: fixed, with a test that would have caught it
- False positive: declined in writing, with evidence
- Partly right: the correct half fixed, and the declined half explained
Where to put the repro
Write repros in a scratch area, never in the worktree that's being gated. A live repro test sitting in the real test suite contaminates the run. Once the bug is fixed, the repro can become a proper regression test.
What changes
With this rule in place, review findings stop being opinions. The reviewer agent spends its effort on attacks it can prove, the implementer spends no time on phantom bugs, and the human only looks at findings that are already demonstrated.
It's the difference between an AI reviewer that sounds smart and one that actually makes the code better. This rule is one part of the multi-agent pipeline I use to ship production code.