What is hermetic state in AI agent tests?

Hermetic state means that each test case of an AI agent runs as its own user, with a starting state that the case declares and nothing else. No memory passes from one case to the next, so the pass rate measures the agent and not the order in which the tests ran.

How it works

An agent with memory remembers earlier conversations. That is what it is built to do. When every test runs as one shared test user, the second case begins with what the first one left behind: preferences the user "expressed", facts the agent "learned". Some cases pass because of those leftovers and some fail because of them.

A hermetic case is sealed, the way a jar is sealed. It runs as its own synthetic user. Its starting state is declared in its setup file and is not accumulated from an earlier conversation. Its date is the date the case sets. When the case ends, its effects are cleaned up and its world ends with it. Nothing enters except what the case declares.

Where it shows up in real work

A suite I own on a production system ran every test as the same simulated user. Its pass rate measured the agent tangled up with the order of execution. I gave every case its own user and its own seeded state, and I showed that state to the model that grades the conversations.

The numbers became honest, and a second result came without extra effort. Once order meant nothing, cases could run at the same time. Parallel runs arrived first for whole sets and later for single cases. The full regression stopped being the slowest step of the pipeline and became something that runs on every release.

What it is not

Hermetic state does not mean taking memory away from the agent. In ordinary software shared state is a design smell that is refactored away, but in an agent memory is the product, so it is contained and not deleted. It is also not a technique for speed: parallelism is a consequence of isolation and cannot come before it.

When you need it

You need it as soon as the agent keeps anything between conversations: memory, personalization, history. The price is real but small. Setup files get longer, and each case pays once for declaring its own world. In exchange, debugging a failure no longer requires digging through what ran before it. A test suite exists to answer one question: did this change break the agent? It answers honestly only when each verdict depends on the test, on the agent and on nothing else.

Work and evidence

Evaluation gates for a multi-agent education system (opens in a new tab)

My contribution: Built the evaluation harness, calibrated LLM judging, isolated case state and environment-specific evaluation gates.

What this supports: The public account describes golden cases as code, a calibrated LLM judge and the evaluation harness around them, on a Google Cloud agent stack.

What it does not establish: The figure of 700,000+ students describes the population served by the system. It does not measure simultaneous users, requests per second or personal ownership of the whole platform. Client code and data remain private.

Source reviewed:

Common questions

How do I know if my agent tests share state?

Run the suite in reverse order and then in a shuffled order. If the results change, the cases are inheriting state from each other. A pass rate that depends on the order is not measuring the agent.

Can agent tests run in parallel?

Yes, once the cases are isolated. Cases that share state cannot run at the same time, because their order matters. Isolation comes first and speed second: a suite with shared state that runs in parallel turns predictable couplings into races.

How does a test get a returning user with history?

The case declares that history in its setup file. It does not obtain it by running an earlier conversation first. Anyone can then read the setup and know everything the agent knows at the start.

Does the judge model need to see the test state?

Yes. If a case starts a user with prior history and the judge does not know it, the judge grades against the wrong world and flags the agent for knowing things the user never said. The declared starting state, including any mocked date, is shown to the judge with the conversation.

Open to senior AI engineering roles and contract engagements, remote in the US, Mexico and Latin America.