LLM evaluation is the set of tests that turns "the answers look fine" into a verdict a pipeline can act on. For an agent, the thing under test is a conversation with tools and state, where the same input can produce many acceptable answers and many subtly wrong ones. A company needs this work when changes to prompts, tools or models ship because a few conversations read well, and nobody can show that the last change did not break something three turns into a conversation.
What I build
- Golden cases written as code. Scripted scenarios with their starting state, the turns of the user, the expected operations and the outcomes that are not acceptable, versioned next to the code they test.
- Deterministic checks. Tool calls, required fields, exact amounts and forbidden operations are checked by ordinary code. A judge never scores what a string comparison can decide.
- A calibrated judge. A second model scores only what needs judgment. Its rubric is corrected against turns labeled by hand, and each judged check is decided by majority vote across several runs.
- A sealed world for each case. Every case runs as its own synthetic user, with its own seeded state and date, and the judge sees that starting state too.
- Gates by environment. A smoke set on every integration merge and the full regression before a promotion.
Signs that an evaluation cannot be trusted yet
- The same change passes one day and fails the next, and people rerun it until it turns green.
- Every case runs as the same test user, so the pass rate moves when the order of the cases changes.
- A model grades what code could check: a format, a number, whether a tool was called.
- Nobody has compared the verdicts of the judge with those of a person, case by case.
Each sign has a known fix, and the order matters. Isolation comes before parallel runs: a suite whose cases share state turns flaky when they run at the same time. Calibration comes before the gate: a gate on a judge nobody checked teaches the team to stop reading its failures.
How the work runs
- Read how a change ships today. I look at the path from a pull request to production and at the failures that already had a cost. Those become the first cases.
- Write the cases. Ordinary tasks, permission limits, ambiguous requests and failures of a dependency. Each case declares its own world: the date, the history of the user, the catalog the agent consults.
- Split the expectations. What code can decide goes to code, and what needs interpretation goes to the judge. Mixing the two is the root of most flaky agent suites.
- Calibrate the judge. We label a set of conversations by hand, run the judge over them and read every disagreement. Sometimes the human label is the wrong one. The corrected rubric lives in version control next to the tests.
- Break the agent on purpose. A forbidden tool call, a wrong price, missing evidence. The suite must fail.
- Connect the gate. The suite runs in the pipeline with the depth that fits each environment, and each result records which candidate was evaluated.
What you receive
The cases, the harness, the rubric with its labeled examples and the pipeline step, in your repository, with notes on how to add a case and how to calibrate again. Calibration does not finish. It converges.
What this does not include
I do not sell a score. A pass rate from a suite that shares state, or from a judge nobody checked, measures the suite. Evaluation does not make a model deterministic either. It makes the things around the model deterministic, so that a no stops the release.
Where the evidence is
For a higher-education group that serves 700,000+ students, I built the machinery that decides whether a change to its production agents may ship: the cases, the calibrated judge, the isolated users and the gates of each environment. The gate replays 138 golden cases before a release. That work surfaced failures worth reading: a judge that reasoned from the real clock and failed correct answers about deadlines, and a judge that reported a formatting defect that was not in the transcript. My research applies the same discipline in public: a pre-registered study of how agents find documents, published with its harness.