An experiment in working with multiple AI models

For the past months, I have been building a side project: a TypeScript monorepo with a mobile app, a web app, and four shared packages. About 22,000 lines of source code plus 7,500 lines of tests, written in large part by AI coding agents under my direction. The feature this story revolves around is backup and restore: users can export everything the app stores about them and import it back later. Code like this tends to fail quietly: a restore can drop half the data without showing an error. That is a level of risk where "the AI said it looks fine" stops being good enough.

Two tools matter for this story. Claude Code is Anthropic's coding agent, which reads the code, plans, edits, and runs the session I work in. Codex is OpenAI's counterpart. For everything in this story, it ran their GPT-5.6 models. Both can also act as reviewers: hand them the code and a brief set of instructions, and they return a list of what they think is wrong. In this story, I never did that handing myself: the Claude Code session I worked in launched every review and every fix run, in both tools. I reviewed the results and made the decisions.

What follows is the story of an experiment. It started with an accident (a paid quota running out) that put two different AI models on the same code and showed me they find different bugs. And it ends with a measured answer to the question the accident raised: when several models look at the same problem, what do you actually get, and how much should you trust any single answer?

To set expectations: this is not a benchmark, not a model ranking, and not a tutorial for a specific tool. The numbers ahead come from one person experimenting on one codebase. They will not tell you which model is "best". They show something I find more useful: how to find out for yourself, and what such an experiment can and cannot claim.

Glossary

A few words carry a specific meaning in this article:

Two model families, one codebase

The backup and restore feature went through review in bundles: I finish a slice of work, and Claude Code packages the changed code with a short brief and dispatches a reviewer model to look at it independently. Dispatched means exactly that: the reviewer runs in its own session, reads the code fresh, and returns a list of findings. It shares no context with the session that wrote the code, and that separation is deliberate. A reviewer sitting inside the author's context tends to inherit the author's blind spots.

For the first review rounds, the reviewer was Codex. Eight rounds produced about thirty findings, and roughly twenty-five of them held up when I checked them against the code. Not one misdescribed what the code actually does. The workflow felt settled: write, dispatch, triage, fix, repeat.

Then my Codex quota ran out mid-session. I still had unreviewed bundles, so I fell back to what was available: Claude Sonnet agents, reviewers from Anthropic's own model family, running the same review instructions on the next bundles. The stand-ins produced eleven findings across four rounds. One of them stood out: the web app's restore could wipe the stored data instead of rolling it back when an import failed halfway through. Not a defect that endangers the app as a whole, but an annoying one: you restore your data and end up with less than you started with. The finding surfaced as a plain side note that a claimed guarantee was untested.

When the quota came back, I had Codex re-review the code the stand-ins had already covered. Sixteen more findings, fourteen of them leading to fixes. And it overturned a decision I had already accepted: an earlier round had approved the way database writes were grouped during restore, and Codex now demonstrated that the app's export could read half-restored data while an import was still running.

They also worked in visibly different ways. The Sonnet reviews showed their evidence: they read the source code of the libraries the app depends on, ran git commands to check their own assumptions, and traced failure paths step by step before claiming anything. Codex asserted its conclusions without showing work, and was mostly right anyway. One caveat on this comparison: the Sonnet agents ran inside my Claude Code setup and had the project's memory files available, the notes the tooling keeps about the codebase, while Codex arrived cold. That muddies any judgment about which reviewer knew the project better, but it does not touch the difference in working style.

The open questions

The review rounds were impressive, but I could not have told you what exactly I had observed, because too much of the setup was uncontrolled.

Start with the reviewer itself. What I called "Codex" was one specific model the whole time: GPT-5.6 comes in three models, sol, terra, and luna, roughly flagship to small, and a configuration file on my machine pinned every review to sol. The pin was deliberate: these models can be told how hard to think, and I wanted reviews at full reasoning rather than a cheaper setting. But it meant no review ever told me anything about the other two models. On top of that, Codex has a memory feature that was switched on, so it may have recognized code from its own earlier sessions rather than reading it fresh. And the Claude stand-ins, as mentioned, had the project's memory files. Each of these is what experimenters call a confound: an alternative explanation for what you saw. Did the stand-ins catch the restore bug because Anthropic models look at code differently, or because they had better notes?

Then there was what I could not see at all. I knew what each reviewer reported. I had no idea what all of them had missed, because nobody knew the full list of bugs in that code. In general, review findings without an answer key tell you about presence, never about absence.

The setup was also fraying operationally. Four reviews died in silence because the connection I had wired Codex through timed out when a review ran long. One review hung for thirty minutes because its brief told Codex not to run shell commands, a well-meant safety instruction that also banned it from reading files, since its file reads are shell commands. And the model PIN I had removed from that configuration file reappeared later without my involvement.

So, I had four questions:

1. The loop: after fixing what a review found, can I hand the reviewer its own earlier session and ask it to re-check, or will it just nod everything through?

2. The models: which of the three GPT-5.6 models is for what, and is the big one actually the best reviewer?

3. The prompt: my review briefs had grown into structured, multi-part instructions. Which parts did the work, and which were superstition?

4. The bill: what does all of this cost, and where is the spend worth it?

To answer them, I designed an experiment.

The experiment setup

The biggest gap was the answer key, so that came first. The earlier review rounds had left me with seven defects in the backup and restore code, each verified by hand and all fixed in a single commit. I reset the code to the state just before that fix and scored every review that followed by counting how many of the seven it reported.

One of the seven made a particularly clean test case. A comment in the database code had stopped matching the code it described:

// Simplified. The comment promises one transaction function://// "the restore clears the table and rewrites it, both inside the// caller's withTransactionAsync, so clear-then-rewrite is atomic"//// The actual caller, in another file, uses a different one:await database.withExclusiveTransactionAsync(async (txn) => { // ... clear and rewrite ...});

The two functions differ in failure behavior, so the comment makes a promise the code no longer keeps. Comments are notes for humans, and a wrong note eventually gets trusted by someone instead of the code. For a reviewer, a mismatch like this works as a canary for careful reading: it can only be spotted by comparing the comment with the code.

The rest of the setup was about making sure a reviewer could only find those bugs the honest way. The commit containing the fixes was parked outside the visible history, so a reviewer browsing git could not stumble on the answers. The two files in which I had documented the defects moved out of the repository for the duration. Codex's memory feature was switched off for every run. Every setting travelled explicitly with every run, model, reasoning effort, and permissions included, after the configuration drift from the previous section. And reviews ran in a read-only mode, so a reviewer could not modify what it was judging.

Then Claude Code ran the plan: seventeen runs over two days, one after another, every output and every token count logged verbatim. The loop question got the first runs: run a focused review of one area, apply one fix, resume the reviewer's session. The model question got a broad review with an identical prompt across sol, terra, and luna. The prompt question got ablations, the same review re-run with one part of the brief removed at a time. The implementer runs let models fix bugs rather than find them. None of it needed special tooling: a git repository and a text file for the log.

What the models found

In the broad review, Sol found three of the seven. Terra also found three, but not the same three. Luna found one.

The focused runs and severity labels filled in the profiles:

• Terra found the comment canary in five out of five runs, even with a deliberately minimal brief, yet across six runs it reported not a single finding about test quality.

• Test-quality findings came only from sol and luna.

• Luna alone read the source code of installed libraries, the packages a project pulls in rather than code I wrote, and its unique confirmed findings sat exactly in that class, at roughly double the runtime and token count.

• Sol marked every finding as blocking, its most urgent label, which makes those labels useless for ordering work. Luna's labels matched reality; Terra's were usable.

The default assumption, take the biggest model for everything, did not survive the data.

Repetition produced its own result: the defect that required reading installed library source turned up in exactly one of six otherwise identical runs. A single run, whichever way it lands, gives no way to tell luck from thoroughness.

In the implementer runs, the models got a set of fix instructions that turned out to contradict themselves: the description said the change applied everywhere, the attached file list was shorter. Terra re-derived the actual scope and fixed all affected places. Sol and Luna followed the list to the letter, a quarter faster. All three reported honestly what they had and had not verified: every claim about tests and checks held up when Claude Code re-ran them afterwards.

The prompt ablations answered the superstition question:

• Without the output contract, the part that fixes the report format, the model invented its own severity scheme and padded the report with reassurance.

• Without the list of already-settled findings, things earlier reviews had raised and I had deliberately left as they are, models re-flagged them. With the list: zero re-flags.

• Without the write fence, the part that limits which files a model may touch, implementer runs made no rogue edits but cost roughly double, including one unrequested whole-file style cleanup.

• A stripped-down brief still caught the canary and nothing subtler.

• No measurable effect: the pointer at my coding-standards files, the part I would have defended hardest.

Combining models: synergy and cost

The loop question had the friendliest answer. After a fix round, handing the reviewer its own earlier session worked: the resumed model noticed the one fix Claude Code had applied, stood by the findings that were still open, and did not wave anything through. It re-checked the actual code rather than trusting its memory of it. The price made it the standard move: about 42,000 tokens per re-validation, against 137,000 to 312,000 for a fresh review of the same scope.

The model results turned into routing rather than a ranking:

In general, the value of several models did not come from headcount. Two runs of the same model mostly retraced each other. The value came from perspectives that differ in shape, a reviewer that reads library internals next to one that reads comments closely, plus rules for combining what they return. The rules I ended with are few: no load-bearing result counts until a second, independent worker has confirmed it. Confirmation crosses the family line where possible, an Anthropic worker checking OpenAI output and the other way round. When two workers disagree, a third one looks, and I decide.

The cross-checking also answers the circularity worry, the suspicion that AI reviewing AI just recycles one model's blind spots through another. The families disagreed where it mattered: one overturned a decision the other had let stand. Circular reviewing would have produced agreement.

The bill question turned out to have two dimensions. The first is the rate: providers charge per token, at different prices per model. The second is efficiency: how many tokens a model needs for the same job. Both move the bill, and often in opposite directions: Luna's broad review burned about twice the tokens of sol's and still cost less, because its rate is a fifth of Sol's. A model at half the price per token that needs three times the tokens is the more expensive model.

For a single model, this stays simple arithmetic: rate times tokens, compare. The verdict from this experiment, use several models together, removes that simplicity. What needs evaluating is no longer model against model but combination against combination: who reviews, who confirms, in which order, for which kind of task. Each pairing also exists in the reverse order, and each behaves differently per scenario. That combinatorial problem fuelled the experiments more than any single question did.

Limitations

Everything above deserves its counterweight, because the list of what this experiment cannot claim is long:

• Most conditions ran exactly once. The repeated condition showed what a single run can hide, and every unrepeated number here carries that uncertainty.

• Every run used maximum reasoning effort. Whether cheaper settings change the picture is unmeasured.

• The code had already survived four review rounds before the experiment. A score against seven defects in hardened code is a floor, not a rate, and says nothing about untouched code.

• The real-usage rounds and the experiment are not cleanly comparable: the rounds ran with Codex's memory feature on and no settled-findings list, the experiment ran memory-off with the list. Differences between the two have more than one explanation.

• One codebase, one feature area, two days of runs, one person. No research team, no statistics, no established benchmark framework, no blind procedure beyond hiding the answers from the models.

So the routing rules from the previous section are working defaults, not laws. What makes them usable anyway is that every number arrived with its conditions attached: when the next model version ships, I know exactly which runs to repeat, and anyone with a git repository can run the same check against their own code before believing mine.

From findings to an executable skill

A pile of verified findings still has a failure mode: me. Rules that live in a strategy document depend on the person applying them, and an AI working session has the same weakness, since after enough hours the early decisions fade from its context. Claude Code has a mechanism that fits better: skills, instruction files the agent loads automatically when a matching task comes up, so the results became one.

The skill activates whenever implementation or review work goes through dispatched workers, and it pins my session to the role it already played throughout this story: the orchestrator. The session owns the task, the triage and the final word on disputes, and it stops writing code itself once workers do. The rest is the previous sections' findings, enforced instead of remembered:

• A pre-flight before any dispatch: the skill checks that Codex is installed and signed in, and a Codex-less run needs my explicit consent rather than a silent fallback.

• A stated plan before the first worker runs: checkpoints, assigned models, how each result gets confirmed. The routing and confirmation rules apply as written.

• Write fences for workers, and one rule the experiment earned directly: a result confirmed by two independent workers is final. The orchestrator does not re-check it, which keeps the session's attention on decisions instead of re-verification.

• Deviations are announced, not absorbed: a failed dispatch, a scope surprise or a disputed result changes the plan visibly.

The skill is public, in the subagent-orchestrator plugin of my it-bens/ai-tools collection on GitHub, together with the distilled experiment notes. When one of the model families revs, the routing table can be re-measured instead of re-guessed.

The general pattern

None of this method is specific to code review, or to these two vendors. It transfers to any situation where you hand work to AI tools and wonder how much to trust the answers:

• Build a case where you know the right answer before you ask. Fixed bugs you can un-fix are one source. Any settled question works.

• Control what the tool can see, so it can only succeed the honest way.

• Run the comparison you care about, and repeat at least one condition.

• Write down the limits next to the results. The numbers stay useful exactly as far as their conditions hold.

• Turn what survives into something that executes on its own rather than into a document.

If you work with a single AI assistant and the stakes are low, none of this is worth the setup. When several models work on the same task, or when a single "looks good" would make you ship something you cannot easily check, an afternoon of building an answer key pays for itself.

Key takeaways

1. No single AI model saw the whole picture: each found a different subset of seven known defects, and the standout catch of the review rounds came from the second model family, brought in by accident.

2. Single impressive runs are misleading: one defect turned up in only one of six identical runs. Without repetition or a second reviewer, a thorough review and a lucky one look the same.

3. AI tools can be measured the way software is tested: freeze a codebase with known defects, plant a canary, control contamination, score the results. No lab, no benchmark framework, and still better than intuition, as long as the limits are named.

4. Model personalities are real and complementary: the flagship was not simply better, one model labelled every finding critical, and the cheapest one read library internals no other model touched.

5. Cost has two dimensions, price per token and tokens needed for the job: a cheaper rate with triple the tokens is the more expensive model. Once several models work together, the evaluation shifts from comparing models to comparing model combinations.

6. Evidence pays off when it becomes executable: the findings ended as a skill the orchestrating agent follows, so the knowledge outlives the person who ran the experiments.

If you rerun any of this on your own code and models, I am interested in where your numbers land, and where they contradict mine. Issues and suggestions are welcome in the plugin repository.

About the author