Storage-compute separation turns engine choice into a reversible decision

Storage-compute separation turns engine choice into a reversible decision—but only if the separation survives contact with metadata, governance, and operations. Putting Parquet files in object storage is not enough. Reversibility comes from owning durable table state outside any one compute engine and keeping every hidden dependency visible.

The architectural appeal is strong. Data can remain in S3, GCS, or ADLS while Spark handles large transformations, Trino serves federated SQL, Flink processes streams, and DuckDB supports local exploration. Compute becomes ephemeral and independently scaled. A team can choose an engine for a workload without paying for a full data migration each time the workload changes.

Open table formats make this safe by adding transactions, snapshots, schema evolution, partition metadata, and time travel over object files. The catalog maps stable table names to current metadata and mediates discovery and access. Together they separate logical tables from the process that reads them. The lakehouse becomes a durable substrate precisely because agents and analytics engines can share governed records without sharing one runtime.

Yet portability leaks quickly. Engines interpret types differently, support different update semantics, optimize different partition layouts, and lag on table-format features. SQL dialects, user-defined functions, materialized views, statistics, row-level policies, and orchestration hooks can all bind a workload to one implementation. The bytes remain open while the operating model becomes proprietary.

I therefore evaluate reversibility as a tested capability. Keep canonical schemas and business semantics outside engine-specific code. Use conformance queries for nulls, timestamps, decimals, nested types, deletes, and snapshot reads. Record which engine wrote each snapshot and verify that another supported engine can read it. Arrow and Parquet provide the byte-level bridge, but table and semantic contracts must complete it.

Maintenance is part of the contract too. Compaction, clustering, vacuum, statistics, and manifest rewriting determine whether a supposedly portable table performs acceptably. If only one engine can maintain the layout it created, switching readers does not remove operational dependence. Ownership for these jobs should sit with the platform, with format-level evidence and recovery procedures rather than opaque vendor automation.

The catalog is the sharper lock-in boundary. It can vend short-lived credentials, enforce row and column policy, track lineage, and host semantic definitions. Those are valuable control-plane functions, but they make migration harder than copying table metadata. Platform consolidation buys governance by spending portability; that trade should be explicit rather than hidden behind the word “open.”

There is one precise concession: a deeply integrated engine is often the right choice when one workload dominates and its specialized optimizer or governance plane creates more value than optionality. Portability has a carrying cost in conformance tests, abstraction, and constrained feature use. The mistake is not accepting lock-in; it is claiming reversibility while depending on features that have never been exercised elsewhere.

I treat engine replacement like disaster recovery: define the boundary, preserve the assets, and rehearse the path. Choose engines by query shape and operating model, then prove that durable data, table history, access semantics, and essential queries survive the choice. Storage-compute separation creates the option. Only tested contracts keep it real. A quarterly portability drill can be small: read one governed snapshot with a second engine, execute representative queries, compare semantics and cost, and record every manual exception. The exceptions are the actual lock-in inventory, and the drill is evidence that the option still exists.