Keep in the context of an agent only what every task needs, and leave everything else where a search can find it and fetch it when a question asks for it. Store a new memory only when something will read it again: in my own measurements, search reached the right document more often than a curated index, and saved memory files were almost never read back.
What agent memory is
The memory of an AI agent is all the information it can reach while it works. The context window is the text the model sees in one call. It is the visible part of memory, and only one part of it.
The rest sits in tiers, much like the RAM, disk and network of a computer.
- Loaded at the start. The instruction files, the catalog of tools and any memory index. This tier is always in view, and it costs tokens on every call.
- Searchable files. Documents and code that the agent can find with a search and then open. This tier is large and cheap, and unseen until a query reaches it.
- Stores behind tools. Databases and APIs. The agent reaches them only by calling a tool, so it has to know that the tool exists.
- Live sources. Information that changes too fast for any stored copy to stay faithful.
Retrieval is the movement of content from the lower tiers into the context window.
How agent memory and retrieval work
Five policies govern the tiers: what loads at the start, what is fetched on demand, which facts are promoted to durable memory, what is removed or compacted, and what checks that a stored copy is still true. I took these names from operating systems on purpose. In the paper they are working vocabulary, not a thesis that I defend.
Content reaches the window in two ways that the agent controls. It can follow a pointer that is already loaded, such as a line in an index that names a file. Or it can search the content directly, with no pointer at all.
The analogy with a computer stops working at one point. A computer copies a memory page exactly. A one-line digest is a different thing: it loses detail when it is written, and it moves further from the document each time the document changes. An agent trusts a stale digest in the same way it trusts a fresh one.
Agent memory and the context window: what is the difference
The context window works like working memory. A small project fits in it completely. When the project grows past the window, the assistant sees fragments.
I met this limit with coding assistants. As a codebase grew, the suggestions got worse, and the assistant forgot parts of the code that mattered. What worked for me was to divide the system into parts small enough to fit in the window, with clear boundaries and explicit interfaces. Each agent then has its own window for one part.
A larger window delays this problem. It does not decide which text deserves the space.
What to store and what to fetch: decision rules
- If every task needs it, keep it loaded and keep it short. Rules and the tool catalog belong here.
- If it is text that a search can reach, leave it findable. Let the agent search and then read the whole document. Do not summarize it first.
- If it lives in a database or behind an API, expose it as a tool with a narrow contract. The guide on connecting a database securely using MCP covers that boundary.
- If it changes constantly, fetch it at the moment of the question and do not store a copy.
- If you write a summary or an index entry, name the check that keeps it true and when that check runs. A digest that nothing validates drifts from its source without any signal.
- If the agent writes notes for future sessions, measure how often a later session opens them. A write that nothing reads is not memory.
What to store from a conversation
I am testing a design that turns conversations into memory. Audio becomes a transcript. An interpretation step maps the transcript onto definitions that I set, such as what counts as a commitment, and it produces structured claims with confidence scores, not summaries. A reconciliation step checks the new claims against what is already recorded.
The rule I keep from it is to store two ledgers: one for what was said and one for what it means. The two stay linked, so every interpretation traces back to its evidence. A transcript alone preserves the words and not what they mean for the person. This is an experiment, and I have no measured result from it.
How to design agent memory step by step
- List the questions the agent must answer, and where each answer lives today.
- Sort each source into a tier: loaded at the start, searchable files, behind a tool or live.
- Reduce the loaded tier to what every task needs.
- Give the agent a search tool over the files, and let it open the documents it finds.
- Freeze a set of questions for which you know the source document.
- Record, for each question, whether the agent opened that document. An accuracy score alone cannot tell you whether the agent failed to find the document or failed to understand it.
- Before you author an index, run the same questions with search alone and compare.
- If the agent writes memories, log every later read of each saved file.
- Write your predictions down before you measure.
What goes wrong with agent memory and retrieval
- The index sends the agent to the wrong file. In my study, the curated policy made 47 wrong-stops, which are answers given without ever reading the correct document. In 43 of them it had read a wrong file. It was not stopping early on thin summaries. It read more and found less.
- Saved memories are not read again. In the same research program, files promoted into a durable memory directory were read later in 2 of 157 eligible cases. A saved fact is useful only when a later read happens.
- Search misses documents because of ranking. In the study, search still missed a fifth of the corpus. The companion study Finding More, Fusing Less found that ranking explained most of that ceiling: ranking with BM25 raised localization by 21.7 points. The fusion default that had shipped failed its own frozen prediction and was retired.
A worked example: a curated index against search on the same questions
The study is Reading More, Finding Less. I froze 102 questions over a production corpus and froze the predictions before any data were read. One policy had no search tool. Its only way to locate a document was an authored index of one-line digests, and the digests were written by a model. The second policy used grep over the corpus and read the files that it returned.
With the index as its only locator, the agent reached the correct document on 52.0% of the questions. With grep, it did so on 80.4%. Where the index did route correctly, the agent answered correctly 86.8% of the time, and a difficulty-matched control showed that the deficit was localization, not comprehension. Downstream, the policy that searched and then read was correct on 72.5% of the questions, and the curated policy on 47.1%.
I then ran a pre-registered public replication over 141 releasable documents and 120 frozen questions, under criteria that were deliberately harder. Search came out 12.5 points ahead on accuracy. Under a symmetric rule, the wrong-stop rate was 34.2% for the curated policy and 18.3% for search. Localization was 62.5% for the index and 75.0% for search. The replication was adjudicated as "revised": one of its hardened predictions, the one about token headroom, failed. The paper reports that failure at the same prominence as the passes. The replication ships its corpus, questions, harness and adjudicator, so anyone can run it again.
What the evidence on agent memory does not show
The routing result is scoped to its substrate. The index was authored, it held one-line digests, the corpus was prose and the index was the only locator. I make no claim about an index used alongside search, which I did not test, or about the accuracy margin on other corpora. The read-back result covers the registered population of files and the channel that was measured.
The conversation design is an experiment without a result. The account of the context window is my own experience with coding assistants and not a controlled measurement, so I give no figure from it here.
The practical reading is not "never curate" and not "delete the memory directory". It is to measure, on your own corpus, whether the index you were about to write beats the search tool the agent already has.
Where to go next
- What is AI agent memory?
- Reading More, Finding Less: the paper, with its predictions, verdicts and replication.
- RAG and data platforms for companies
- Enterprise AI agent architecture: corporate data, security and scale