The Model Writes, the Judge Measures: Anatomy of an LLM Judge

Documentation

Listen to full article~21 min

Listen to AudioSpotlight~3 min

The Model Writes, the Judge Measures: Anatomy of an LLM judge

By Sam Wen, CEO & Founder of XInfer.AI — July 23, 2026

Here is the kind of sentence that should keep a team up at night:

"At this clarity grade, you won't see a thing with the naked eye."

Our assistant produced it while explaining a diamond. The grade it cites is real — SI1, printed on the stone's lab report. The promise attached to it is invented. A clarity grade is assigned under 10× magnification; it does not resolve what a naked eye will see. Two SI1 stones can look entirely different — one clean at arm's length, the other with a visible speck right in the middle. The model took a true fact and manufactured its consequence, because that is what a helpful sentence sounds like.

Notice what the sentence is not. It is not a banned word. It is not a hallucinated stone. It contains no wrong number. It breaks no rule you could have written down in advance. It is simply fluent — and fluency is the problem. Fluent and true are different variables, and a language model optimizes the first one.

We build AI assistants for exactly this kind of product. The one in this essay explains diamonds: a short, two-voice spoken conversation about one stone — a host asks the questions a shopper would ask, a gemologist answers — generated from that stone's evidence. A diamond is the definition of a high-consideration purchase: thousands of dollars, bought rarely, chosen under uncertainty. And it has a property that makes it an unforgiving test bed for generative AI — nearly every claim about a diamond can be checked against its certificate from GIA or IGI, the independent laboratories that grade the world's diamonds. Overselling is not just tacky here. It is checkable.

In an earlier essay we argued that the language model should conduct the conversation while a deterministic core makes every decision. This essay is the same worldview applied to quality. The model writes. It does not get to grade.

At one generation you can catch a sentence like the one above by reading it. Across a catalogue of stones you cannot. So before we made the writer better, we built an instrument: an independent LLM judge that reads every generated claim against the evidence and returns a verdict we can count. This essay is about how that judge is built, how we keep the judge itself honest, and what changed once we had it — which is that the arguments stopped.

Part I — Three gates, and who owns which rule

The assistant explains one diamond at a time, from that diamond's evidence: the lab certificate, the price, the retailer's published analysis of the stone. One rule governs everything: every claim about this stone must trace to supplied evidence. If the evidence cannot answer — is it eye-clean, meaning no marks visible to the naked eye? — the assistant says so, out loud: the honest answer is a maybe, and it will not promise more. Then it points the shopper to a human gemologist.

Enforcing that rule is not one mechanism. It is three, layered, and they differ in kind.

The first gate prevents. It runs in code, inside the pipeline. When the assistant generates a follow-up question about a stone, the question must name the evidence path it would be answered from. A question that names no available path is rejected — by code, not by prompt hope. This is what stops the system from asking "how does this price compare to the market?" about a stone whose data has no comparable set. The model never gets the chance to improvise an answer, because the question never survives.

The second gate catches. A deterministic validator runs over every finished script: a hard-fail lexicon of the words the assistant must never say — amazing, must-have, investment, the manufactured urgency, the "trust me." Word-boundary regexes, a few dozen tests, no judgment involved. If a banned phrase appears, the script fails. Cheap, fast, and absolute.

The third gate judges. What remains after prevention and regex is the dangerous class: the plausible-but-unsupported claim. "You won't see a thing with the naked eye" passes both gates. No rule names it in advance, because it is not a phrase — it is a relationship between a sentence and a lab report. Checking it requires reading both. That is a job for a reader, and at scale the reader has to be a model.

One detail here taught us more than we expected: the boundary between the gates is itself a design decision. When we extended the banned-word list from the domain experts' style guide, the unambiguous terms went into the regex gate. But the context-dependent ones deliberately did not — and the best example is flawless. In most products, "flawless" is empty praise. In diamonds it is literally a grade: FL, the top of the GIA clarity scale. "It is not flawless" is a factual statement about an SI1 stone, not a compliment withheld. Likewise "guaranteed": a real test fixture proved that "that would need a guarantee the report doesn't give" is the assistant at its most honest. A blunt regex would fail exactly those sentences. Those words went to the judge instead, which can read the sense and not just the string. Which gate owns which rule is not a default. It is a decision, and we made it with test evidence.

Part II — Anatomy of a verdict

"Use an LLM judge" usually means: ask a model to rate the output from one to ten. A score like that is an opinion with a number attached. You cannot appeal it, you cannot audit it, and you cannot tell whether a 7 means two small problems or one large one. It is not an instrument; it is a mood.

Our judge is built differently, and every difference exists because a naive version failed first.

It returns a triage, not a score. For every sentence that asserts something beyond restating the input data, the judge answers typed questions: Is this a claim about this stone, or about diamonds in general? If it is about this stone — does the evidence establish it? If it is general — is it accurate gemology? "Lab-grown and natural diamonds are chemically identical" is category teaching: permitted, and wanted — the judge's instructions say teaching is never a violation. "This stone will look white to the eye" is an item claim, and it had better trace to something. A real violation is only the intersection: about this stone, and unsupported.

A claim also has an owner. Supported is not the only axis. GIA graded the stone; the retailer analyzed it; the assistant speaks for the house. If it says "we grade this VS1," every word is factually true and the sentence is still a violation — it credits the lab's work to the house. The judge flags misattribution as its own class, separate from unsupported claims, because provenance errors are how a trustworthy voice quietly claims authority it does not have.

It must report the claims it cleared. The judge is required to list the general statements and the supported claims too, with the classification visible — and it is forbidden from letting them inflate the violation count. This sounds like bureaucracy. It is the whole point. A verdict that only lists problems is a number you have to trust. A verdict that shows every ruling — each quote verbatim, each one naming the evidence path it was checked against — is an audit trail you can re-check. When the judge is wrong, you can see exactly where.

Orthogonal signals stay orthogonal. The judge also measures tone: does the assistant sound like a trusted expert across the counter or a closer working a commission? Does the conversation sound like two people responding to a stone, or a script? But tone is a separate, subordinate channel, and the judge's instructions are blunt: warmth never excuses an unsupported claim. A formatting check rides in a third channel and changes nothing in the honesty count. There is no blended score, deliberately. A blended score is where warmth hides dishonesty.

The writer and the judge read the same constitution. Every rule about what a lab report does and does not support lives in one content-standard document. That document is spliced into the writer's prompt and into the judge's prompt. The spec is a single artifact: the writer implements it, the judge enforces it, and editing one sentence changes both sides of the contract at once. There is no version of the rules that only the judge believes.

It never grades its own output. The writers run on Claude Sonnet 5. The judge's seat has been held by two models — first GPT-5.5, a different lab entirely, and today Claude Opus 4.8, a different model from the writers — and we verified the discipline held across the change. A model asked to grade its own writing has a conflict of interest baked into its weights. Independence has to be constructed.

It never touches production. The judge is a development instrument, not a runtime guardrail. It grades output during the dev cycle to find where the writer's prompts leak; it adds zero latency and zero cost to serving. Grading one diamond costs about thirty cents — four passes, each re-reading the full evidence — which is trivially cheap in development and pointlessly expensive in production, where the deterministic gates already stand. A judge is not a content filter. It is how you find out what to fix.

Part III — Calibrate the instrument

The uncomfortable part: the judge is also a language model. It is an unreliable narrator hired to audit an unreliable narrator. This is less absurd than it sounds — auditors are people too — but only if the workflow plans for it. Ours does, in three ways.

The judge can be wrong, and the loop says so. Our written dev cycle is: the judge finds a violation → we fix the writer's prompt, or we fix the judge if the judge is wrong → we log it → we re-run. The same humility applies to the deterministic gates. We ban the word flaw — the marks inside a stone are characteristics, never flaws. Then the assistant, explaining a modest luster rating, said it was "not a flaw; just part of why the rating sits at four instead of five or six" — and the regex failed the most honest sentence in the script. So the ban was narrowed: the corrective "not a flaw" now passes, while "no flaws in this stone" — a perfection claim — still fails. The exemption got its own tests. False positives are not noise; they are calibration data.

The verdict is schema-checked, and malformed verdicts retry. In real runs the judge would intermittently drop a required field, or return a field with the wrong type. We do not average around that; the harness detects the malformed verdict and retries it, and every consumer of the verdict is guarded against a bad shape. An instrument with unhandled failure modes doesn't stop measuring — it starts lying quietly.

One sample is noise. The same stone, judged twice, does not score the same. We watched one diamond score fifteen violations on one run and six to eight on reruns. If we had compared two models using single samples, we could have concluded anything at all. The discipline is boring: average at least two runs before believing any comparison, and never let a single verdict decide anything important.

An instrument you never calibrate is just another opinion.

Part IV — Three arguments we never had to have

Here is what actually changed when the judge existed. Three decisions that AI teams normally settle by taste, seniority, or vendor reputation got settled by reading the same number before and after.

Which model. We ran two writer tiers over the same stones, with the same prompts and the same judge. Claude Haiku 4.5 — the small, fast tier — averaged about ten real violations per diamond. Claude Sonnet 5 averaged 0.8. A twelve-fold honesty gap, on identical inputs. And the direction of the tradeoff surprised us: Sonnet also scored higher on specificity, so the honesty did not cost richness. Every arithmetic blunder in the corpus came from the small model —

[truncated for AI cost control]