Three months ago we made a rule: no PR merges until someone runs the diff through the Review a PR Diff for Real Bugs and Security Holes (Skip the Nits) prompt and pastes the output into the thread. I was the team's resident skeptic of process-for-process's-sake — I have killed more useless checklists than I've created — so I refused to just declare victory. I did the thing I should always do and logged data instead of vibes for a full quarter. Here's the unglamorous, honest retrospective, including the parts that didn't flatter the tool.
What we measured
For every single finding we recorded three things: severity, category, and a verdict assigned after the fact — real defect, valid-but-minor, or false positive. No grading on a curve. Across 91 PRs the prompt produced 214 findings. Roughly 60% were MEDIUM or LOW, and most of those were legitimately worth a glance even when they weren't blockers. The CRITICAL-and-HIGH bucket held 38 findings, and 14 of those were confirmed genuine defects we reproduced in testing — bugs that would otherwise have shipped. The number that genuinely surprised me, the one I went back and re-checked because I didn't believe it: zero false positives at the CRITICAL level over three months. Every CRITICAL was real. The model was conservative about that top tier in exactly the way the prompt's 'do not invent findings to look thorough' rule asks it to be, which is the single behavior that makes a tool like this trustworthy instead of exhausting.
What it changed about how we work
The process effects turned out bigger than the bug-catching, and most of them were second-order things I didn't predict when I grumpily agreed to the experiment.
- Human review turnaround dropped about 30% — people stopped re-litigating the obvious correctness stuff the AI had already cleared and spent their finite attention on design, naming, and product fit instead
- Two genuine production regressions were caught pre-merge that we'd budgeted, in hindsight, at four-plus debugging hours each had they reached prod — that alone paid for the whole quarter of friction
- Junior devs started reading the 'Why it matters' column like security flashcards — one told me they finally understood why a timing-safe comparison matters from a single finding on a token-check
- It caught a 'this silently changes behavior for existing callers' issue the prompt specifically watches for, which our feature-focused human reviews would never have framed as a risk because the feature itself worked fine
The friction, honestly
It is not free and I won't pretend otherwise. Pasting a large diff and waiting ten seconds is a real step in the flow, and on a one-line typo-fix PR it feels faintly ridiculous, so we exempt anything under twenty lines. The MEDIUM-tier false positives — almost always 'possible N+1' on a query that's genuinely fine at our scale, or a flagged exception catch that's intentional — taught the team an important habit: read the findings as prompts for thought, not as gospel to obey. The first month, a junior dev 'fixed' three non-problems because the AI mentioned them, and we had to explicitly coach 'the table is an agenda, not a verdict.' And the deepest limitation is unfixable by better prompting: it has no product judgment. It will give a clean bill of health to a feature that should never have been built, because it reviews the code, not the decision.
Would we keep it
Yes, with zero hesitation, and the skeptic who logged the data is the one saying so. Ten seconds of friction weighed against one 3am incident is not a close call; it's not even the same order of magnitude. We're now wiring it into a GitHub Action so the paste happens automatically and the review lands as a PR comment, removing the one bit of friction that was ours to remove. One thing I'd tell anyone starting: socialize the 'findings are an agenda, not a verdict' rule on day one, before a junior dev dutifully 'fixes' three non-problems and concludes the tool is dumb. The value compounds once the team learns to argue with it.
If you want the same data on your own team instead of trusting mine, grab the Review a PR Diff for Real Bugs and Security Holes (Skip the Nits) prompt on Prompt Dock, log your findings for a month, and decide on numbers instead of opinions — then feed every confirmed defect straight into my Write a Failing Test That Reproduces This Bug Before I Fix It prompt so the same bug can never bill you twice.