Arrow and Parquet are the lingua franca of the AI data stack
The composability everyone praises in the modern data stack is not an architecture achievement — it is a byte-layout agreement. Arrow defines what columnar data looks like in memory; Parquet defines what it looks like on disk; and because nearly every engine that matters speaks both, you can compose Spark, DuckDB, Polars, DataFusion, and a Python notebook into one pipeline without a single serialization boundary doing violence to your data. When I evaluate a data stack for AI workloads now, my first question isn't which engines it runs. It's where the format boundaries are.
The strategic feature is zero-copy. Arrow's C Data Interface is a C ABI, not a library: any runtime that implements it can hand record batches to any other with zero bytes copied — Python to Rust, Java to R, no serialization tax at any hop. That is what made Arrow the interchange format rather than yet another DataFrame internal. The same logic runs over the wire: Arrow Flight moves record batches between systems at throughput JDBC was never designed for, and ADBC is quietly replacing row-oriented connectivity with columnar-native access. Every hop that used to cost a serialize-deserialize round trip now costs a pointer handoff.
On disk, Parquet won so completely that the interesting arguments moved up a layer. Delta Lake, Iceberg, and Hudi — three table formats fighting a genuine standards war — all agreed without ceremony on Parquet as the data file underneath. The pruning machinery is why: row-group statistics skip whole megabyte-scale chunks whose min/max ranges exclude the predicate, bloom filters answer equality checks without touching the column chunk, and page-level statistics prune within what remains. A well-laid-out Parquet dataset can answer a selective query reading under 1% of its physical bytes. For AI workloads — feature extraction, embedding pipelines, eval-set scans, agents querying a lakehouse — that pruning pipeline is the difference between interactive and batch.
The consequence for architecture is that engines become interchangeable parts. If your pipeline's contract is "Arrow record batches in memory, Parquet files under an open table format at rest," then the choice between Spark and DuckDB, or Polars and Pandas, is a per-stage sizing decision instead of a platform commitment. I've stopped designing pipelines around vendor APIs and started designing them around format boundaries — the stable interfaces are the byte layouts, because those are the ones every vendor has already agreed to. That's also the honest reading of lock-in: a platform that stores your data as open Parquet with an open metadata layer can charge you for compute, but it cannot hold your bytes hostage.
The concession: the lingua franca is not frozen, and treating today's bytes as an endpoint would be its own mistake. Vortex — now under Linux Foundation governance — packages a decade of encoding research (FastLanes, ALP, FSST) into a columnar format that outperforms Parquet on modern hardware, and Parquet itself keeps absorbing new types and encodings. The stable bet is the interface, not the implementation: open columnar formats with statistics-driven pruning and zero-copy interchange. If Vortex or a Parquet 3 displaces today's bytes, it will do so by speaking Arrow at the boundary — which is precisely the point.
What I ask of any AI data platform now: where do bytes get copied, and who decided the file layout? If the answer to the first is "at every tool boundary" and the second is "a proprietary engine," the platform is fighting the decade. The stack that survives is the one where memory means Arrow, disk means Parquet-under-a-table-format, and everything else is a replaceable engine choice.