Designing memory for agent systems

Most discussion of agent memory jumps straight to mechanism — vector stores, embeddings, retrieval pipelines. That's premature. Agent memory is a data modeling problem wearing an AI costume, and the modeling decisions dominate the mechanism decisions. Two years of memory-system churn have only strengthened this claim: the products that survived — Letta's tiered core/recall/archival design, Cognee's extract-cognify-load pipeline over pluggable graph and vector backends, the hand-rolled Postgres-plus-pgvector facts tables that quietly outnumber both — differ in mechanism but agree on the part that matters. They all converged on typed, structured records. The schema was the point all along.

Three questions before any mechanism

1. What is a memory? A fact, a preference, an episode, a skill? Each has a different lifecycle. A preference can be updated in place; an episode is append-only; a fact can be falsified and must be deletable. Collapsing them into one undifferentiated "memory" table is the original sin of most agent memory designs.
2. Who owns each fact? When the same information lives in long-term memory and in working context, you have duplicated state — and every lesson from database normalization applies. Duplication is fine; unowned duplication is how agents confidently act on stale beliefs.
3. What's the forgetting policy? A memory system without deletion is a liability, not an asset. Staleness, contradiction, and privacy all demand the ability to retract — which means memories need provenance: where the belief came from and when it was last confirmed.

A schema sketch

The minimal viable structure I keep arriving at is typed records with provenance:

memory:
  type:        fact | preference | episode | reference
  content:     the assertion itself
  source:      where this came from (conversation, document, inference)
  confirmed:   when it was last validated
  links:       related memories (the garden pattern, again)

Retrieval quality then becomes mostly a function of how well-typed and well-linked the records are. Embeddings help you find candidates; structure tells you whether to trust them. The current systems sort into three camps by where they put that structure — persistence-first designs that model memory as records, structure-first designs that model it as a knowledge graph, and orchestrator-coupled designs that inherit whatever the framework's state object happens to be. The third camp is the one I steer clients away from: coupling your memory schema to this year's orchestrator is how you end up migrating beliefs instead of code.

One thing I under-weighted when I first wrote this note: time belongs in the schema twice. Organizational memory systems now ship bi-temporal records — when a fact was true versus when the agent learned it — which is precisely the distinction dimensional modelers have handled with slowly changing dimensions for decades. An agent that can't distinguish "the customer's limit was ₹5 lakh when we decided" from "we recorded the new limit on Tuesday" can't explain its own past actions, and for anything operating under audit obligations that's disqualifying, not inconvenient.

Open threads

Consolidation — when do many episodes become one fact? — has moved from open question to design pattern: the newer pipelines run an explicit post-ingestion maintenance stage that prunes, reweights, and promotes episodic records into semantic ones, which is to say memory now has a scheduled maintenance job, like every other database worth trusting. The working-context budget went the way I suspected: deciding what earns a place in the prompt is cache eviction, and the eviction-policy literature applies almost unchanged. What remains genuinely open for me is cross-agent memory — when a fleet shares an organizational memory graph, write contention and trust in another agent's beliefs turn into distributed-systems problems that the single-agent framing above doesn't answer.