At some point in every AI agent project, someone proposes the obvious idea: use a second AI model to grade the first one. Run the conversation, hand the transcript to the grader, get a score, and let that score decide whether the change ships. People call this grader a judge. The idea is right. The first implementation is almost always wrong, and wrong in a way that is worse than having no quality gate at all.
Here is why. An unreliable judge fails good changes and passes bad ones, at random, on its own schedule. Engineers respond the way engineers always respond to a flaky check: they re-run it until it goes green, and they stop reading the failures. Within a month the gate is theater. You pay the full cost of an evaluation system, you get none of the protection, and you carry a false sense that "we have evals."
A judge earns the right to block a release the same way a person would. It has to show, case by case, that its judgment matches a standard you trust. That process is called calibration. On a production system serving hundreds of thousands of users, it was the most valuable work I did on our testing setup. It was also unglamorous, specific, and full of surprises. Three of those surprises changed how I build these systems.
The judge lived in the wrong time
Good test cases control everything about their world, including the date. A test about an enrollment deadline fixes a specific "today" so its expected answers stay stable forever. The judge, meanwhile, was quietly reasoning from the real clock. So it failed correct answers about deadlines because, from where it stood, those deadlines had already passed. The agent was right. The test was right. The verdict was wrong. That is the worst kind of failure, because it erodes trust in the exact tool that is supposed to produce trust.
The fix was simple: tell the judge what day the test believes it is, right next to the transcript. Obvious in hindsight. But notice the shape of the bug. The judge depended on the real world in a way the rest of the test system had carefully eliminated. The judge is part of the test system. Every rule you apply to your tests applies to the judge too.
The judge invented a defect
During one calibration round, the judge reported that the agent's response contained stray formatting characters. It did not. The judge, which is itself a language model, had invented the very defect it was grading.
No instructions fix this. The lesson is categorical: never ask a judge to score anything that plain code can check. Formatting rules, exact numbers, whether a tool was called, whether a required phrase appears: all of that moved into ordinary code with ordinary assertions. Code cannot hallucinate. The judge kept only the questions that genuinely need judgment: did the answer address the question, was it correct in substance, was the tone right. Shrinking the judge's territory is not a defeat. It is the point. Every check you move from the model into code is a check that can never lie to you again.
One verdict is an anecdote
Ask the judge once and you get an opinion. Ask on Tuesday and again on Wednesday and you may get two different opinions. A release gate cannot flip on an unlucky sample in either direction. A false no teaches people to ignore the gate. A false yes is a production incident with a paper trail saying everything passed.
So judged checks vote. We run the judge several times on the same conversation and take the majority verdict. It costs more, and it is worth every token, because a stable verdict is the entire product. A gate that flickers is not a gate.
What calibration actually is
The work itself is not clever. Grade a set of conversations by hand. Run the judge over the same set. Find every disagreement between the judge and the human. For each one, decide who was wrong. Sometimes the human label is the wrong one. Then fix the written grading instructions until judge and human agree. Keep those instructions in version control next to the tests, so a change to what "good" means gets reviewed like any code change.
Two closing observations from doing this on a real system.
First, calibration never finishes. It converges. Each new capability produces new ways to fail, and each of those either becomes a labeled example the judge scores correctly, or a plain-code check the judge no longer touches. The ratchet turns one way: the judge's territory gets smaller, and its accuracy inside that territory gets higher.
Second, the part I would tell anyone starting: the purpose of a judge is not to score conversations. It is to make a no mean something. The day your team stops arguing with the gate, because the gate has been right often enough that arguing is not worth it, is the day you have an evaluation system instead of an evaluation ritual.