Agent memory over a lakehouse

A reference build that turns this site's central thesis — agent memory is a data-modeling problem, not a retrieval afterthought — into a working architecture. No client, no benchmark numbers. The deliverable is the pattern, not a score.

The problem

Most agent "memory" is a vector store with a prompt. You embed everything, search it, stuff the top results into context, and call it memory. That works until the first time you need to know what the system believed when it made a decision, or to make the system forget something — a user, a contract, a fact that changed — and discover the store has no notion of either.

The failure is a category error. Retrieval answers "what is similar to this query." Memory answers "what do we know, when did we learn it, and under what conditions may we use it." A vector store only does the first. Building memory on top of one means provenance, versioning, and forgetting all have to be bolted on afterward — which is precisely when they quietly stop happening.

The constraints

This is a reference build, so the constraints are chosen to mirror what an enterprise engagement actually looks like rather than a demo. Memory has to be typed and governed from day one. The system has to answer an auditor's question — who acted, on what belief, with what authority — without a special export. It has to support forgetting as a first-class operation, not a cleanup script. And it has to sit on a data foundation that already carries the governance, lineage, and retention controls an enterprise has invested in, because an agent is only as trustworthy as the data it acts on.

The decisions

Four of them, each a rejection of an easier default.

Memory is typed records, not embeddings. The store holds fact, preference, episode, and reference as distinct record types with distinct lifecycles, not one undifferentiated embedding blob. Facts are slowly-changing; preferences expire and conflict; episodes are append-only; references are immutable citations. The vector index is a read accelerator layered on top of this — the way an index is layered on a table — never the system of record. This is writes as schema decisions, reads as query plans made load-bearing: you write through a typed schema, and retrieval becomes one possible query plan rather than the definition of memory.

Memory is scoped, not flat. Type is only one axis. A production memory store also holds memory at the level of the organization, the division, the team, and the individual customer or actor interaction — and those are not the same kind of thing. An organization's compliance posture changes on a governance cycle; a customer's preference changes in a single conversation. They have different writers, different access boundaries, and different forgetting rules. Collapsing them into one undifferentiated store forces a choice between leaking one tenant's memory into another and inventing namespace prefixes the retrieval layer then ignores. So scope is a schema axis in its own right: every record carries what it is about and who it belongs to, and that scope — not the type — is what access control and retention policy key on.

Versioning is bi-temporal, not overwrite-in-place. The lakehouse already solves slowly-changing dimensions for data warehouses; the build reuses that machinery for memory. Every fact carries when it became true and when the system learned it, so "what did we believe when we decided X" is a query, not a forensic reconstruction. Rejecting the overwrite default is what makes audit and rollback cheap instead of impossible.

Governance before embeddings. Access, retention, and provenance are properties of the record schema, enforced by the platform — not prompts reminding a model to be careful. A memory the model can read past its own permission is a data leak, and the fix is architectural, not instructional. This is memory needs governance before it needs embeddings, made concrete.

The outcome

The build answers the two questions a production memory system has to answer and a vector store cannot: what is true now, and what did we believe when we decided. Audit trails fall out of the schema rather than being generated after the fact. Forgetting is a retention policy enforced at the data layer, not a one-off deletion. Session state and long-term memory are separate schemas with separate retention — so a conversation's scratch space never silently becomes the system's knowledge.

The honest caveat: this is a reference build, not a deployed system with production traffic. There are no latency percentiles or recall curves to show, because the point of the build is the architecture — that memory is a data-modeling discipline, and that the lakehouse is its natural substrate — not a number to quote. A number would tell you it's fast or accurate; the pattern tells you whether it can be trusted, governed, and audited over years. For a system that has to survive contact with regulation and time, that's the claim that matters.

Reference implementation

The pattern is instantiated as a runnable companion library — DuckDB + Delta Lake, zero infrastructure. Typed records with distinct lifecycles; an org → division → team → actor scope hierarchy; bi-temporal reads where valid-time is valid_from and system-time is the Delta transaction log, so what did we believe when we decided is a time-travel read, not a hand-rolled history table; and governance — scope-filtered access, tombstone-then-purge forgetting, and an audit trail reconstructed from the commit log. The semantic index is rebuilt from the store on demand, so forgetting a record removes it from the index too: the index is never the system of record.

The library is the product; the agent loop is a thin adapter over it. A behavioral eval suite asserts the four properties — recall, provenance, forgetting, and scope isolation — and a demo drives a local model through the memory layer to show the semantics live in the store, not the model.

The full source — library, evals, and six self-checking demo scenarios — is public at github.com/Dhristhi/agent-memory-lakehouse, under the Apache-2.0 license.