What is a golden case in LLM evaluation?

A golden case is a scripted scenario for an AI agent: a starting state, the turns of the user and the outcome expected at each turn. It is kept as code next to the agent, and the whole set is replayed before a release to see whether a change broke something.

How it works

A golden case has four parts: the initial state, the turns the user will say, the operations the agent is expected to perform and the outcomes that are not acceptable. A small example: a user asks for the invoice of another customer. The expected result is a denial, and no protected record may come back.

A useful set covers ordinary tasks, authorization boundaries, ambiguous requests and failures of the systems the agent depends on. The cases are versioned together with the configuration they evaluate.

Inside a case, the expectations are of two kinds. Some are decided by plain code: was the scheduling tool called, did the agent refuse to invent a price. Others need judgment: did the answer respond to the request, in the right register, with the right facts about the program. Mixing the two kinds is the root of most unstable agent suites.

Where it shows up in real work

For the agents of a higher-education group I wrote the golden cases as code, in the repository of the agent. A prospective student asks about a program. An enrolled student needs to reschedule. A conversation has to end in a handoff to a person. Each case writes down what is expected at every turn. The date, the situation of the student and the catalog the agent consults come from the fixture of the case, never from the outside world. Today the gate replays 138 golden cases before a release.

What it is not

A golden case is not a permanent truth: it records what the product should do under the current policy, and when the policy changes the expectation is reviewed with it. It is also not material for tuning. Keep a separate held-out set for wider validation, so that adjusting a rubric does not turn into memorizing the test set.

When you need them

Write golden cases when the agent makes decisions whose silent change would matter: a price, an appointment, a handoff. Reading a few conversations by hand and feeling good about them works for a while. It stops working with the first change that breaks the appointment logic three turns into a conversation that nobody read again. From that point the scenarios have to be written down, with their expectations, and replayed on every change.

Work and evidence

Evaluation gates for a multi-agent education system (opens in a new tab)

My contribution: Built the evaluation harness, calibrated LLM judging, isolated case state and environment-specific evaluation gates.

What this supports: The public account describes golden cases as code, a calibrated LLM judge and the evaluation harness around them, on a Google Cloud agent stack.

What it does not establish: The figure of 700,000+ students describes the population served by the system. It does not measure simultaneous users, requests per second or personal ownership of the whole platform. Client code and data remain private.

Source reviewed:

Common questions

How many golden cases do I need?

There is no number I would give as a rule. In my work two properties mattered more than the count: every case carries its own mocked state, and every case separates what code can check from what needs judgment. For coverage, include ordinary tasks, authorization boundaries, ambiguous requests and dependency failures.

Where should golden cases be stored?

In the repository, next to the code and the configuration they test. A change to the behavior of the agent and the change to its expectations then arrive in one pull request and are reviewed together.

What is the difference between a golden case and a unit test?

A unit test makes deterministic assertions over deterministic code. A golden case tests a conversation, where one input can produce many acceptable phrasings and many subtly wrong ones. So it combines exact checks in code with expectations that a calibrated judge model scores.

What happens to a golden case when the product changes?

The expectation is reviewed together with the policy that changed. A golden case is a reference expectation, not an immutable truth, so updating it is normal work. Because the case is code, the update is reviewed like any other change.

Open to senior AI engineering roles and contract engagements, remote in the US, Mexico and Latin America.