Agent memory over a lakehouse

Open full-size diagram ↗

The problem

Most agent "memory" is a vector store with a prompt. You embed everything, search it, stuff the top results into context, and call it memory. That works until the first time you need to know what the system believed when it made a decision, or to make the system forget something — a user, a contract, a fact that changed — and discover the store has no notion of either.

The failure is a category error. Retrieval answers "what is similar to this query." Memory answers "what do we know, when did we learn it, and under what conditions may we use it." A vector store only does the first. Building memory on top of one means provenance, versioning, and forgetting all have to be bolted on afterward — which is precisely when they quietly stop happening.

The constraints

This is a reference build, so the constraints are chosen to mirror what an enterprise engagement actually looks like rather than a demo. Memory has to be typed and governed from day one. The system has to answer an auditor's question — who acted, on what belief, with what authority — without a special export. It has to support forgetting as a first-class operation, not a cleanup script. And it has to sit on a data foundation that already carries the governance, lineage, and retention controls an enterprise has invested in, because an agent is only as trustworthy as the data it acts on.

The decisions

Four of them, each a rejection of an easier default.

Memory is typed records, not embeddings. The store holds fact, preference, episode, and reference as distinct record types with distinct lifecycles, not one undifferentiated embedding blob. Facts are slowly-changing; preferences expire and conflict; episodes are append-only; references are immutable citations. The vector index is a read accelerator layered on top of this — the way an index is layered on a table — never the system of record. This is writes as schema decisions, reads as query plans made load-bearing: you write through a typed schema, and retrieval becomes one possible query plan rather than the definition of memory.

Memory is scoped, not flat. Type is only one axis. A production memory store also holds memory at the level of the organization, the division, the team, and the individual customer or actor interaction — and those are not the same kind of thing. An organization's compliance posture changes on a governance cycle; a customer's preference changes in a single conversation. They have different writers, different access boundaries, and different forgetting rules. Collapsing them into one undifferentiated store forces a choice between leaking one tenant's memory into another and inventing namespace prefixes the retrieval layer then ignores. So scope is a schema axis in its own right: every record carries what it is about and who it belongs to, and that scope — not the type — is what access control and retention policy key on.

Versioning is bi-temporal, not overwrite-in-place. The lakehouse already solves slowly-changing dimensions for data warehouses; the build reuses that machinery for memory. Every fact carries when it became true and when the system learned it, so "what did we believe when we decided X" is a query, not a forensic reconstruction. Rejecting the overwrite default is what makes audit and rollback cheap instead of impossible.

Governance before embeddings. Access, retention, and provenance are properties of the record schema, enforced by the platform — not prompts reminding a model to be careful. A memory the model can read past its own permission is a data leak, and the fix is architectural, not instructional. This is memory needs governance before it needs embeddings, made concrete.

The outcome

In the checked reference build, the store answers two questions a production memory system has to preserve: what is true now, and what did we believe when we decided. The transaction log supplies the audit history. Forgetting is implemented as tombstone followed by purge rather than a one-off deletion, while true storage reclamation still requires Delta VACUUM. Session state and long-term memory remain separate schemas with separate retention — so a conversation's scratch space does not silently become the system's knowledge.

The honest caveat: this is a locally verified reference build, not a deployed system with production traffic. There are no latency percentiles, recall curves, long-running retention results, or client outcomes. The evidence supports the bounded implementation described here; it does not prove that the pattern is fast, compliant, secure, or reliable at production scale.

Reference implementation

The pattern is instantiated as a runnable companion library — DuckDB + Delta Lake, zero infrastructure. Typed records with distinct lifecycles; an org → division → team → actor scope hierarchy; bi-temporal reads where valid-time is valid_from and system-time is the Delta transaction log, so what did we believe when we decided is a time-travel read, not a hand-rolled history table; and governance — scope-filtered access, tombstone-then-purge forgetting, and an audit trail reconstructed from the commit log. The semantic index is rebuilt from the store on demand, so forgetting a record removes it from the index too: the index is never the system of record.

The library is the product; the agent loop is a thin adapter over it. A behavioral eval suite asserts the four properties — recall, provenance, forgetting, and scope isolation — and a demo drives a local model through the memory layer to show the semantics live in the store, not the model.

The full source — library, evals, and six self-checking demo scenarios — is public at github.com/Dhristhi/agent-memory-lakehouse, under the Apache-2.0 license.

What you can run

git clone https://github.com/Dhristhi/agent-memory-lakehouse
cd agent-memory-lakehouse
uv sync --extra dev
uv run python -m pytest -q
uv run python -m agent_loop.demo
for s in scripts/scenario_*.py; do uv run python "$s"; done