The lakehouse is the natural substrate for agent memory

Write down what an agent memory system actually needs from its storage layer, and an odd thing happens: you've written the feature list of an open table format. Cheap, append-heavy writes for a stream of episodic events. Time travel, because "what did the agent believe on Tuesday?" is an audit question you will be asked. Schema evolution, because the shape of what agents record changes every time you revise a prompt or add a tool. Atomic merge, because consolidating new observations into a profile is an upsert, not an append. And multi-engine access, because the process that writes memories is never the only process that reads them.

Delta Lake, Iceberg, and Hudi were built to deliver exactly this property set — ACID transactions, schema evolution, time travel, and efficient upserts — by layering metadata management over Parquet on object storage. Iceberg gives you FOR SYSTEM_TIME AS OF queries, snapshot isolation, and version rollback: an agent's memory state at any historical point, reconstructable on demand, with a bad consolidation run reversible by resetting the snapshot pointer. Delta gives you atomic MERGE for match-update-insert consolidation in one pass, and a Change Data Feed that turns the memory store itself into a CDC stream — new memories can trigger downstream re-embedding or summarisation without a separate queue. Iceberg's schema evolution uses stable field IDs rather than column positions, so adding a "confidence" field to episodic records doesn't rewrite terabytes of history.

The multi-engine argument

The deeper fit is architectural. The lakehouse's defining choice is storage/compute separation: data lives on object storage, engines are ephemeral. The catalog layer makes multi-engine access safe — every engine sees a consistent snapshot of the table. That matters for memory because memory has at least three consumers with little in common: the agent runtime doing retrieval, the batch job doing consolidation, and the analyst asking why the agent behaved strangely last week. On a lakehouse that can be one governed table read by DuckDB for local inspection, Spark for consolidation, and Trino for forensics, without an export step or a second copy drifting out of sync.

I would not stop at a table format, though. The catalog becomes part of the memory control plane. It can vend short-lived storage credentials, apply row and column policy, preserve snapshot identity, and expose lineage to every engine. That is valuable when agent fan-out turns one human query into many machine reads. It also names the lock-in honestly: Delta, Iceberg, and Hudi are converging at the storage layer while catalogs still diverge in governance, semantics, and maintenance. Open files make compute replaceable; they do not make policy or operating practice portable.

The honest counter-case is latency. Object storage is not a hot path. An agent that needs working memory inside a turn cannot wait on a Parquet scan through S3; the active session belongs in a key-value store or an in-process cache. The lakehouse earns its place one tier down, as the episodic, analytical, and archival record behind that serving tier. Streaming small appends also resurrects the small-file problem, so compaction, retention, snapshot expiry, and index refresh become explicit operating jobs rather than background magic.

Which is the point. The "agent memory platform" category is being sold as new infrastructure, and the retrieval layer above the store genuinely is new — what to write and what to recall is an unsolved design problem. But the storage substrate underneath is a solved one, and it's likely already running in your organisation, governed, backed up, and staffed. There's also a convergence dividend: as agent-era data goes streaming-first, the same substrate absorbs both the event firehose and the memory it condenses into. Before buying a memory database, ask your data team what table format they run. The answer is probably also your agent's long-term memory.