A test suite for an AI agent can look very reassuring: dozens of scripted conversations, each with expected outcomes, all green. Nobody asks the question that matters while everything is green: green for whom? If your agent has memory, and every production agent has memory, the answer might be "green for the one test user all your cases share, in the one order they happen to run."
That was the state of a suite I own on a production system. Every test ran as the same simulated user. The agent's memory did what memory is supposed to do: it remembered. Test B started with leftovers from test A. Preferences the user had "expressed." Facts the agent had "learned." Context that had no business existing in test B's world. Some tests passed because of those leftovers. Some failed because of them. The pass rate was not measuring the agent. It was measuring the agent tangled up with the order the tests happened to run in.
Why this problem is specific to AI agents
Every engineer knows the old rule: tests must not share state. In ordinary software, shared state is a design smell you refactor away. In an agent, the shared state is the product. Memory, personalization, continuity between conversations: the things that make the agent good in production are exactly the things that poison the tests. You cannot delete memory. You have to contain it.
The containment is simple to state: every test runs as its own user, with its own starting state, in its own sealed world. Nothing enters the test except what the test itself declares. The date is the test's date. The user's history is the test's history. When the test ends, its world ends with it. I call these hermetic cases, sealed the way a jar is sealed.
Two implementation details carried most of the value.
Starting state is declared, not accumulated. A test that needs a returning user with history does not get that history by running an earlier conversation first. It declares the history in its setup file. That sounds like extra work, and it is, once. In exchange, every test is self-contained: you can read its setup and know everything the agent knows. Debugging stopped requiring archaeology into what ran before.
The grader sees the starting state too. We use an AI judge to grade these conversations. If a test starts a user off with prior history and the judge does not know that, the judge grades against the wrong world. It flags the agent for "knowing things the user never said." So the declared starting state is shown to the judge along with the conversation. Isolation applies to the whole testing path, not just the agent.
The bonus: the tests can finally run in parallel
Here is the part that turned a correctness fix into a speed fix. Tests that share state cannot run at the same time, because order matters. So the suite runs one test after another, gets slower as it grows, and slides toward the fate of all slow suites: run nightly, then weekly, then only when someone remembers.
The moment every test carried its own world, order stopped meaning anything, and running tests at the same time required no cleverness at all. The full test run went from the slowest step in our pipeline to something we run on every release without thinking about it.
The order of operations matters, and it generalizes: isolation first, speed second. If you parallelize a suite whose tests share state, you do not get a faster suite. You get a flaky one, because couplings that were at least predictable one at a time become races. Every hour spent on isolation was secretly also an hour spent on speed. I just did not know it at the time.
The uncomfortable question
If your agent's test suite has never been through this, the honest question is not "are my tests isolated?" It is "what is my pass rate actually measuring?" Run your suite in reverse order. Run it shuffled. If the results change, you do not have an evaluation system yet. You have a very expensive way of measuring your execution order.
The fix is not glamorous. Setup files get longer. Every test pays the cost of declaring its own world. But a test suite exists to answer one question: did this change break the agent? It can only answer honestly when each verdict depends on the test, the agent, and nothing else. That "nothing else" is what hermetic cases buy.