There is a moment in every agent project where someone proposes the obvious idea: use a second model to grade the first one. Run the conversation, hand the transcript to a judge, get a score, gate the release on it. The idea is right. The implementation, the first time, is almost always wrong — and wrong in a way that is worse than having no gate at all.
Here is why. An uncalibrated judge fails good changes and passes bad ones, at random, on a schedule of its own. Engineers respond the way engineers always respond to a flaky check: they re-run it until it goes green, and they stop reading the failures. Within a month the gate is theater. You have all the cost of an evaluation system and none of the protection, plus a false sense that "we have evals."
A judge earns the right to block a release the same way a person would: by demonstrating, case by case, that its judgment matches a standard you trust. That process is calibration, and on a production agent system — one serving hundreds of thousands of users — it was the single highest-leverage work I did on the evaluation harness. It was also unglamorous, specific, and full of surprises. Three of them changed how I build these systems.
The judge lived in the wrong time
Good golden cases mock their state, and that includes the date. A case about an enrollment deadline seeds a specific today so its expectations are stable forever. The judge, meanwhile, was quietly reasoning from the real clock. Result: it failed correct answers about deadlines because, from where it stood, those deadlines were in the past. The agent was right, the case was right, and the verdict was wrong — the worst kind of failure, because it erodes trust in the exact tool that is supposed to produce trust.
The fix was to anchor the judge's temporal reasoning to the case's mocked today, injected alongside the transcript. Obvious in retrospect. But notice the shape of the bug: the judge had an implicit dependency on ambient reality that the rest of the harness had carefully eliminated. Judges are part of the test system, and every discipline you apply to fixtures applies to them too.
The judge hallucinated a defect
During one calibration round, the judge reported that the agent's response contained stray markdown formatting. It did not. The judge — a language model, after all — had invented the defect it was grading.
You cannot rubric your way out of this one. The lesson is categorical: a judge should never be asked to score what a string comparison can decide. Formatting rules, exact figures, whether a tool was called, whether a phrase that must appear appears — all of that moved to a deterministic-only mode, plain code with plain assertions. The judge's jurisdiction shrank to the things that genuinely need judgment: was the answer responsive, correct in substance, right in register. Shrinking a judge's jurisdiction is not a defeat. It is the calibration working — every check you move from the model to code is a check that can no longer hallucinate.
One verdict is an anecdote
Even a calibrated judge samples. Ask it once and you get an opinion; ask it once on Tuesday and once on Wednesday and you may get two. A release gate cannot flip on an unlucky sample in either direction — a false no teaches people to ignore the gate, a false yes is a production incident with a paper trail saying it passed.
So judged checks vote: multiple judge runs, majority verdict. It costs more tokens, and it is worth every one, because the stability of the verdict is the entire product. A gate that flickers is not a gate.
What calibration actually is
The work itself was not clever. Hand-label a set of turns. Run the judge over them. Find every disagreement between judge and human. For each one, decide who was wrong — sometimes it was the human label — and fix the rubric, not the model, until agreement holds. Then hold the rubric under version control next to the cases, so a change to what "good" means gets reviewed like the code change it is.
Two closing observations from doing this on a real system.
First, calibration never finishes; it converges. New capabilities produce new failure modes, and each one either becomes a labeled example the judge scores correctly, or a deterministic check the judge no longer touches. The ratchet only turns one way: over time the judge's jurisdiction gets smaller and its accuracy inside that jurisdiction gets higher.
Second, and this is the part I would tell anyone starting: the purpose of a judge is not to score conversations. It is to let a no mean something. The day your team merges without arguing with the gate — because the gate has been right often enough that arguing stopped being worth it — is the day you have an evaluation system instead of an evaluation ritual.