Tokenization changes are data migrations, not model upgrades

Tokenization changes are data migrations, not model upgrades. I treat a change in how text becomes token IDs as a change to the representation contract of the system. The source document may be identical while its stored token sequence, measured length, and token-based boundaries change. Calling this a model refresh hides the work: identify dependent artifacts, regenerate what is incompatible, and prove that the application still receives the evidence it needs.

The mechanism matters. Byte pair encoding learns merge rules from recurring patterns, turning byte sequences into vocabulary entries. Those rules, the vocabulary, and any pre-splitting rules determine how text is segmented. An integer token ID is meaningful within that encoding, not a universal identifier for a word. I would no more move an unlabelled token array between encodings than move database keys between unrelated lookup tables and assume the numbers still mean the same thing.

That makes source preservation my first requirement. I want retained text and structured messages to remain the basis for regeneration, with derived token arrays carrying their encoding identity. A package version alone is not the contract; I need to know which vocabulary and rules produced the artifact. Where preprocessing transforms the source before encoding, that transformation needs an identity too. Otherwise a later comparison cannot distinguish changed source, changed preprocessing, and changed tokenization.

The less obvious migration is in the decisions made from token counts. A chunker that cuts documents at a token limit may produce different passages under another encoding. A prompt assembler with the same numerical budget may admit different evidence or truncate at a different point. An answer can then change without any retrieval-ranking change at all. I would trace these dependencies before rebuilding anything: token-based chunks, length filters, prompt admission checks, and truncation policies are consumers of the representation.

Conversation formatting belongs in that inventory. Messages are structured records, but a model receives a sequence in which delimiters and role markers express that structure. Different chat protocols use different representations, and the formatting itself consumes context. Counting only the visible message text misses part of the input. I therefore keep the logical conversation separate from its model-specific serialization and test the complete assembled request. A tokenizer change and a chat-template change are distinct changes that can arrive in the same release.

My acceptance test would start with a fixed source sample spanning the actual workload: prose, code, identifiers, whitespace variations, and the languages the application supports. I would compare old and new token counts, inspect changed chunk boundaries, and record which passages survive prompt assembly. Round-trip decoding checks representation integrity; downstream task tests check whether the retained evidence remains sufficient. A lower average token count is useful evidence about packing, not proof that every important request improved.

For rollout, I would keep old and new derived artifacts distinguishable rather than overwrite them under the same identity. The affected pipeline should be reproducible from retained source, and rollback should restore compatible encoding, serialization, and budget decisions together. I would also retain the encoding identity beside historical token measurements. Otherwise a usage trend can mix a change in workload with a change in its unit of measurement, leaving a cost review to explain an apparent efficiency gain it has not actually established.

There is a precise boundary to this argument: a stateless application that sends raw text to a hosted model, persists no token-derived artifacts, and uses no local token-based admission or chunking rules may have nothing to backfill. Its obligation is compatibility and budget testing rather than a stored-data migration. I would not build migration machinery where there is no dependent state.

Where that state exists, I want the release question to change. Not merely whether the new model answers better, but which representations became incompatible, which evidence-selection decisions changed, and how the previous behavior can be restored. Tokenization is upstream of what the model gets to see. Its migration deserves an owner before the model upgrade gets a launch date.