Schema evolution is producer-consumer negotiation, not parser permissiveness

Schema evolution is producer-consumer negotiation, not parser permissiveness. A platform has not made change safe merely because a writer can append a column or a reader can ignore one. Evolution is safe only when the producer states the change, affected consumers can evaluate it, and both sides have an enforceable path through the transition.

Storage engines make permissiveness easy. Delta can add columns during writes or merges, widen selected types, and decouple logical names from physical columns. These mechanisms are valuable, but they answer “can the table represent this change?” rather than “can the system still mean what its consumers think it means?” A nullable field may be mechanically additive while changing a metric denominator. An integer widened to a decimal may preserve values while changing hash behavior. A renamed column may be metadata-only for storage and still break a downstream SQL model.

Data contracts are the API layer of the data platform because they put meaning, ownership, quality, and service expectations around the schema. Semantic versioning makes the producer declare compatibility rather than infer it from a successful write. Major versions identify breaking change; minor versions cover compatible additions; conformance tests check the artifact before deployment. The important move is not the numbering convention — it is making compatibility an explicit claim that can be challenged.

Negotiation also needs impact analysis. A catalog and lineage graph should identify consumers of a changed field, including views, streams, metrics, activation jobs, and model features. Owners need a deprecation window and a way to acknowledge migration. Versioned views and expand-contract migrations let producers add the new shape, dual-publish while consumers move, and remove the old path only after evidence shows it is unused. This costs more than turning on automatic merge, but it moves risk from incident response into planned work.

Enforcement belongs at several boundaries. The write path checks structural compatibility. Expectations check ranges, nullability, uniqueness, and row-level conditions. Contract tests check the published interface. Consumer tests verify interpretation against representative data. Observability turns a violation into an event with an owner and policy, not a surprise embedded in a dashboard three days later. Metadata is the control plane when it can connect those recorded decisions to the assets and people they affect.

This is why global permissive settings are dangerous. A session-wide auto-evolution flag delegates an architectural decision to whichever job happens to write next. Per-operation evolution is better because intent travels with the change. Better still is a release record carrying contract version, compatibility classification, impact set, rollout stage, and rollback path. The table protocol may govern readable bytes; the release protocol must govern readable meaning.

There is one precise concession: append-only exploratory datasets with no declared consumers can use permissive evolution to preserve discovery speed. Once an asset feeds a production decision, shared metric, agent context, or external interface, that boundary disappears. The burden of negotiation should scale with consumer consequence, not be imposed uniformly on every scratch table.

Schema enforcement prevents malformed data; schema evolution permits new shapes. Neither establishes consent. Production data platforms need all three: a storage mechanism that can change, a contract that says what change means, and a migration protocol that lets consumers move on their own clock. Treat evolution as a conversation with evidence, and compatibility becomes managed risk instead of parser luck.