AI Engineering

Notes on making probabilistic systems shippable: evaluation harnesses, LLM judges, context engineering, agents, and the GCP estate underneath them. Everything here comes from production systems — method extracted from work, written to be reused.

2026-08-14 · agents · context-engineering

Memory Lives Beyond the Context Window

An agent's memory is everything it can reach, not just its context window. A preprint, Reading More, Finding Less, tests that idea with pre-registered experiments and finds the failure is routing: a hand-curated memory index sent the agent to the wrong files, plain file search beat it, and saved memories almost never got read again. The full corpus and harness ship with the paper for re-running.

Read →

2026-08-05 · evals · harness · agents

Can You Trust an AI to Grade Your AI?

Many teams use one AI model to grade another one's answers, and then let that grade decide what ships. Almost nobody checks whether the grader itself can be trusted. This is what it took, on a real production system, to make the grader reliable enough that a failing score is allowed to stop a release.

Read →

2026-08-05 · evals · testing · agents

Your AI Agent Passes Its Tests for the Wrong Reason

AI agents are built to remember, and that breaks their tests. When test cases share one user, each case inherits leftover memory from the one before it, so the pass rate measures the order the tests ran in, not the agent. Giving every test its own clean user and its own sealed world fixed that, and let the whole suite run in parallel for free.

Read →

Related: the case study on evaluation gates for a 700,000-student agent system, the GCP dossier, and the working papers on agent harnesses and structural waste.