LLM and AI agent evaluation for companies

I build the evaluation that decides whether a change to an AI agent or an LLM feature may ship: cases written as code, checks in plain code, a calibrated judge and a gate in the pipeline. It is for teams that already have a model in production, or close to it, and cannot say whether the last change made it better or worse.

LLM evaluation is the set of tests that turns "the answers look fine" into a verdict a pipeline can act on. For an agent, the thing under test is a conversation with tools and state, where the same input can produce many acceptable answers and many subtly wrong ones. A company needs this work when changes to prompts, tools or models ship because a few conversations read well, and nobody can show that the last change did not break something three turns into a conversation.

What I build

  • Golden cases written as code. Scripted scenarios with their starting state, the turns of the user, the expected operations and the outcomes that are not acceptable, versioned next to the code they test.
  • Deterministic checks. Tool calls, required fields, exact amounts and forbidden operations are checked by ordinary code. A judge never scores what a string comparison can decide.
  • A calibrated judge. A second model scores only what needs judgment. Its rubric is corrected against turns labeled by hand, and each judged check is decided by majority vote across several runs.
  • A sealed world for each case. Every case runs as its own synthetic user, with its own seeded state and date, and the judge sees that starting state too.
  • Gates by environment. A smoke set on every integration merge and the full regression before a promotion.

Signs that an evaluation cannot be trusted yet

  • The same change passes one day and fails the next, and people rerun it until it turns green.
  • Every case runs as the same test user, so the pass rate moves when the order of the cases changes.
  • A model grades what code could check: a format, a number, whether a tool was called.
  • Nobody has compared the verdicts of the judge with those of a person, case by case.

Each sign has a known fix, and the order matters. Isolation comes before parallel runs: a suite whose cases share state turns flaky when they run at the same time. Calibration comes before the gate: a gate on a judge nobody checked teaches the team to stop reading its failures.

How the work runs

  1. Read how a change ships today. I look at the path from a pull request to production and at the failures that already had a cost. Those become the first cases.
  2. Write the cases. Ordinary tasks, permission limits, ambiguous requests and failures of a dependency. Each case declares its own world: the date, the history of the user, the catalog the agent consults.
  3. Split the expectations. What code can decide goes to code, and what needs interpretation goes to the judge. Mixing the two is the root of most flaky agent suites.
  4. Calibrate the judge. We label a set of conversations by hand, run the judge over them and read every disagreement. Sometimes the human label is the wrong one. The corrected rubric lives in version control next to the tests.
  5. Break the agent on purpose. A forbidden tool call, a wrong price, missing evidence. The suite must fail.
  6. Connect the gate. The suite runs in the pipeline with the depth that fits each environment, and each result records which candidate was evaluated.

What you receive

The cases, the harness, the rubric with its labeled examples and the pipeline step, in your repository, with notes on how to add a case and how to calibrate again. Calibration does not finish. It converges.

What this does not include

I do not sell a score. A pass rate from a suite that shares state, or from a judge nobody checked, measures the suite. Evaluation does not make a model deterministic either. It makes the things around the model deterministic, so that a no stops the release.

Where the evidence is

For a higher-education group that serves 700,000+ students, I built the machinery that decides whether a change to its production agents may ship: the cases, the calibrated judge, the isolated users and the gates of each environment. The gate replays 138 golden cases before a release. That work surfaced failures worth reading: a judge that reasoned from the real clock and failed correct answers about deadlines, and a judge that reported a formatting defect that was not in the transcript. My research applies the same discipline in public: a pre-registered study of how agents find documents, published with its harness.

Work and evidence

Evaluation gates for a multi-agent education system (opens in a new tab)

My contribution: Built the evaluation harness, calibrated LLM judging, isolated case state and environment-specific evaluation gates.

What this supports: The public account describes golden cases as code, a calibrated LLM judge and the evaluation harness around them, on a Google Cloud agent stack.

What it does not establish: The figure of 700,000+ students describes the population served by the system. It does not measure simultaneous users, requests per second or personal ownership of the whole platform. Client code and data remain private.

Source reviewed:

Reading More, Finding Less (opens in a new tab)

My contribution: Authored a pre-registered study and public replication of agent retrieval routing.

What this supports: Evaluation methodology with a released corpus, frozen questions, harness and adjudicator.

What it does not establish: Results concern the studied document corpus and routing policies, not all RAG or agent systems.

Source reviewed:

Common questions

How do you test an AI agent before it goes to production?

With scripted cases that fix the starting state, the turns of the user and the expected operations, checked in two ways: ordinary code for anything exact, and a calibrated judge model for what needs interpretation. The suite runs in the pipeline, and a failing verdict blocks the merge.

Can we trust an LLM to judge another LLM?

Only after it is calibrated. I compare the judge with labels written by people, case by case, and correct the rubric until they agree. Everything code can check is taken away from the judge, and what it does judge is decided by majority vote across several runs, so one unlucky sample cannot flip a release.

We already have evals. Why do they pass one day and fail the next?

Usually for one of three reasons: the cases share state through the memory of the agent, the judge reasons from the real date and not from the date of the case, or a single run of the judge decides the verdict. Running the suite in a different order shows the first one quickly.

Do you work with teams in the United States and Mexico?

Yes. I work remotely from Monterrey, Mexico, in English and in Spanish, on contract or in a senior role. The evaluation lives in your repository and your pipeline, so the work does not need me in your office.

Are you an LLM evaluation consultant, or do you build the evaluation yourself?

I build it. I am a Senior AI Engineer who takes contract work: I write the cases with your team, build the harness and the judge, calibrate it and connect the gate to your pipeline.

Related questions

Open to senior AI engineering roles and contract engagements, remote in the US, Mexico and Latin America.