An LLM judge is a second language model that reads a conversation, together with the expectations written for it, and returns a verdict. It is useful only after calibration: its verdicts are compared with labels written by people until the two agree.
The judge receives the transcript of a conversation, the expectations written in the test case and a rubric. The rubric is the written instruction that says what a good answer is. With those, the judge returns a verdict for each turn it is asked about.
A judge is not trusted on arrival. Calibration earns the trust. Someone grades a set of conversations by hand, and the judge grades the same set. Every disagreement is read and someone decides who was wrong, because sometimes the human label is the wrong one. Then the rubric is corrected, not the model, until judge and person agree. The rubric lives in version control next to the tests, so a change to what "good" means is reviewed like any change to code.
Two more rules keep the verdict stable. The judge is shown the state of the case, including its mocked date. And every judged check runs several times and the majority wins, so one unlucky sample cannot decide a release.
Where it shows up in real work
I calibrated the judge for the agents of a higher-education group that serves 700,000+ students. Three failures taught me the most. The judge reasoned from the real clock while the cases fixed their own date, so it failed correct answers about deadlines. In one round it reported stray formatting characters that were not in the transcript. And a single verdict could change from one run to the next. The fixes were to tell the judge the date of the case, to move everything code can check into code, and to vote.
What it is not
A judge does not replace assertions: formatting, exact amounts, required phrases and whether a tool was called belong in ordinary code, because code cannot hallucinate. It is also not a score to watch on a chart. Its purpose is to make a "no" mean something.
When you need one
You need a judge when part of what makes an answer correct needs interpretation: whether it addressed the question, whether it was right in substance, whether the tone fit. If every expectation can be written as a string comparison, plain code is enough. If the verdict is going to stop a release, plan the calibration first. An uncalibrated judge fails good changes, people run it again until it passes, and the gate stops protecting anything. Calibration also never finishes. It converges, as each new failure becomes a labeled example or a check in code.
My contribution: Built the evaluation harness, calibrated LLM judging, isolated case state and environment-specific evaluation gates.
What this supports: The public account describes golden cases as code, a calibrated LLM judge and the evaluation harness around them, on a Google Cloud agent stack.
What it does not establish: The figure of 700,000+ students describes the population served by the system. It does not measure simultaneous users, requests per second or personal ownership of the whole platform. Client code and data remain private.
Source reviewed:
Common questions
Can an LLM judge replace human review?
No. A judge is calibrated against labels that people wrote, and each new capability of the agent needs new labeled examples or new checks in code. In my work the human QA team kept testing the same behavior alongside the automated evaluation. The judge makes the routine verdict repeatable, and people still decide what a good answer is.
How do you calibrate an LLM judge?
Grade a set of conversations by hand, run the judge over the same set and read every disagreement. Decide who was wrong in each one, then correct the written rubric until judge and person agree. Count false passes and false failures separately, and keep the versions of the model and of the rubric so a change in scores can be traced.
Why does an LLM judge give different verdicts on the same answer?
Because the judge is a language model and it samples. Running it several times and taking the majority reduces that variation. Voting does not repair a rubric that is wrong in a systematic way. Only calibration does that.
What should an LLM judge not grade?
Anything plain code can check: formatting rules, exact numbers, whether a tool was called, whether a required phrase appears. In one calibration round my judge reported a formatting defect that was not in the transcript. Those checks moved to ordinary assertions, and the judge kept only the questions that need judgment.