Automated LLM evaluation turns an expected behavior into a repeatable check. For a production agent, the unit under test is often a conversation with tools and state. A fluent answer can still use the wrong record, invent a price or skip a required handoff. The evaluation needs to inspect those outcomes.
Start with cases that represent decisions
Write scenarios with their initial state, user turns, expected operations and unacceptable outcomes. Include ordinary tasks, authorization boundaries, ambiguous requests and dependency failures. Version the cases alongside the configuration they evaluate.
A golden case is a reference expectation, not an immutable truth. When the product's policy changes, review the expectation with the policy. Keep a held-out set for broader validation so tuning a rubric does not become memorizing the test set.
Use deterministic checks where possible
Tool names, required fields, exact amounts and forbidden operations should be checked directly. Reserve model-based judging for qualities that need interpretation, such as whether a response addresses the user's request in context. Give the judge the actual fixture state, including any mocked date.
Illustrative case:
user asks for another customer's invoice
expected: access denied; no protected record returned
checks: authorization decision + actual tool result
judge: optional assessment of the explanation, not permission enforcement
Calibrate the judge against human labels
Compare verdicts with labeled examples, inspect disagreements and revise the rubric. Measure false passes and false failures separately. Repeated judgments or voting can reduce sampling variance; they do not remove a systematic rubric error. Preserve model and rubric versions so a score change can be investigated.
My public education-system case describes building golden cases as code and the evaluation harness around them. It explains concrete failures: a judge using the wrong date, invented formatting defects and shared state that contaminated cases.
Isolate state and fit the gate to the decision
Give each case its own seeded user and state. Clean up test effects and prevent one case from changing another's starting conditions. Run a small smoke set on frequent integration changes and a fuller regression before promotion. Record which candidate was evaluated; a green result for a different configuration is not release evidence.
Evaluate the evaluator
Deliberately introduce a forbidden tool call, an incorrect price or missing evidence and check that the suite fails. Then restore valid behavior and verify it passes. Watch for flaky verdicts, coverage gaps and cases that pass because the relevant action never ran.
When selecting an engineer, ask to see failed cases, calibration disagreements and how a rejected candidate was prevented from shipping. The work and research below provide starting points for that inspection.