Making AI Code Review Measurable

Code review has always been one of the bottlenecks for software engineers. At loveholidays, we have been using a lot of AI in our daily workflows. Whether a PR is written with the help of AI or not, someone still needs to review it.

Then I read a thread by Intercom on how they use AI not just for code review, but also to approve it. Sounds risky, but also very interesting.

Disclaimer: This post is not an endorsement. This was just an experiment inspired by the thread and my own interpretation of how a system like that might work. It was also only possible because OpenAI gave us a three-month trial. During the process, I burned through a few billion tokens. It was a great learning experience, but definitely not cheap.

1. Why build this myself?#

There are already a lot of code review tools out there. I’m sure some of them are great, and some may even be better than what I built. But I have always had a love and hate relationship with code review tools, even before AI. They are often hit or miss. And when the cost is high, it is hard for me to recommend a tool broadly without first understanding how useful it actually is.

I also wanted something that could run locally inside my existing workflow and use my existing ChatGPT Codex subscription, whether personal or enterprise. I really, really did not want to pay another subscription.

1. Coding agent#

Almost all coding agents have a non-interactive mode. With Codex, you can trigger it via codex exec. I ran all sorts of automation with this approach.

I decided to use codex exec to run the code review. Using only the git diff would be a lot faster and cheaper, but I personally do not like that approach. Context matters a lot.

When I review a pull request, I rarely look at the diff in isolation. I open related files, check how the surrounding code works, look for similar patterns, and think about how the change could break existing behaviour. I wanted the AI reviewer to be able to do the same.

Another reason I liked this approach is that it fits into my existing workflow. The agent can use the same skills, MCP servers, commands, and project context that I already use day to day. That makes the review less like a generic external tool and more like an extension of how I already work.

The basic flow looks like this:

Clone the pull request into a new git worktree location.

git worktree makes it easier to run multiple evaluations in parallel.

Run codex exec with a custom prompt, rules, and output format.

Run this in a sandbox, your own machine, or any environment you are comfortable with, as long as you understand the risks and limitations.

Ask the agent to return a structured JSON output.

Example output:

{ "verdict": "rejected", "reviews": [ { "file": "src/file_1.ts", "line": 42, "comment": "This will break feature A.", "suggestion": "Consider rewriting it this way." } ] }