The lakehouse is the natural substrate for agent memory

Write down what an agent memory system actually needs from its storage layer, and an odd thing happens: you've written the feature list of an open table format. Cheap, append-heavy writes for a stream of episodic events. Time travel, because "what did the agent believe on Tuesday?" is an audit question you will be asked. Schema evolution, because the shape of what agents record changes every time you revise a prompt or add a tool. Atomic merge, because consolidating new observations into a profile is an upsert, not an append. And multi-engine access, because the process that writes memories is never the only process that reads them.

Delta Lake, Iceberg, and Hudi were built to deliver exactly this property set — ACID transactions, schema evolution, time travel, and efficient upserts — by layering metadata management over Parquet on object storage. Iceberg gives you FOR SYSTEM_TIME AS OF queries, snapshot isolation, and version rollback: an agent's memory state at any historical point, reconstructable on demand, with a bad consolidation run reversible by resetting the snapshot pointer. Delta gives you atomic MERGE for match-update-insert consolidation in one pass, and a Change Data Feed that turns the memory store itself into a CDC stream — new memories can trigger downstream re-embedding or summarisation without a separate queue. Iceberg's schema evolution uses stable field IDs rather than column positions, so adding a "confidence" field to episodic records doesn't rewrite terabytes of history.

The multi-engine argument

The deeper fit is architectural. The lakehouse's defining choice is storage/compute separation: data lives on object storage, engines are ephemeral. The catalog layer makes multi-engine access safe — every engine sees a consistent view of the table's current state. That matters for memory because memory has at least three consumer types with nothing in common: the agent runtime doing retrieval, the batch job doing consolidation and summarisation, and the analyst asking why the agent behaved strangely last week. On a lakehouse that's the same table read by different engines — DuckDB for local inspection, Spark for the consolidation pipeline, Trino for ad-hoc forensics — with no export step and no second copy drifting out of sync. Even the medallion pattern maps cleanly: raw interaction events in Bronze, validated and deduplicated episodes in Silver, consolidated profiles and distilled knowledge in Gold.

The honest counter-case is latency. Object storage is not a hot path. An agent that needs a working-memory lookup inside a single turn cannot wait on a Parquet scan through S3; the hot tier — core memory, the active session — belongs in a key-value store or an in-process cache. The lakehouse earns its place one tier down, as the episodic and archival layers: the hot/cold split that memory architectures like MemGPT formalised as memory paging maps naturally onto cache-plus-lakehouse. And streaming small appends into Parquet resurrects the classic small-file problem, so you're signing up for compaction as an operational discipline. This is real work, but it's work your data platform team already knows how to run.

Which is the point. The "agent memory platform" category is being sold as new infrastructure, and the retrieval layer above the store genuinely is new — what to write and what to recall is an unsolved design problem. But the storage substrate underneath is a solved one, and it's likely already running in your organisation, governed, backed up, and staffed. There's also a convergence dividend: as agent-era data goes streaming-first, the same substrate absorbs both the event firehose and the memory it condenses into. Before buying a memory database, ask your data team what table format they run. The answer is probably also your agent's long-term memory.