Continual model updates need knowledge-change ledgers

I want every continually updated model to carry a knowledge-change ledger. A checkpoint identifies an artifact; it does not explain which beliefs the team meant to revise, why those revisions were justified, or whether subsequent updates preserved them. Those are different questions from whether training completed. My release requirement would be a queryable record connecting intended changes to source evidence, serving versions, and observed behavior — not a changelog written after deployment.

The distinction follows from continual-learning and knowledge-editing research. Continual learning addresses acquiring new capabilities while retaining earlier ones. Knowledge editing attempts narrower changes to learned associations. Both encounter interference: learning something new can damage something previously learned, and later edits can weaken earlier corrections. I read that as an operational warning. An accepted update is a claim that needs continuing verification, not a permanent addition to an inventory of facts the model reliably knows.

I would start the ledger with the change request rather than the training job. Each entry would identify the prior claim, its proposed replacement, the supporting source and version, the scope in which it applies, and the person accountable for approval. Effective time should be separate from deployment time. A policy can become valid before the system learns it, or be loaded before it takes effect. That distinction belongs in data, not in an engineer’s recollection.

The application record would then say how the change reached the system: retrieved context, an external editing memory, a weight edit, or continued training. These mechanisms do not make identical promises. A retrieval update changes available evidence; a weight edit changes the model itself. I would link the request to the actual artifacts and serving configuration, preserving failed attempts and superseded decisions. Otherwise the history describes what someone wanted to ship rather than what users could encounter.

Acceptance needs more than the corrected prompt. The editing literature distinguishes direct success from paraphrase robustness, preservation of unrelated knowledge, and propagation into dependent reasoning. MQuAKE makes the last distinction particularly important: answering a revised association does not establish that a chained inference will use it. My ledger would attach tests for the change, its known dependencies, and behavior that must remain stable. An untested dependency should remain visibly untested rather than inherit a green status from its parent.

The same logic applies across releases. I would rerun earlier correction tests alongside retention and behavioral checks, recording results against each candidate version. TRACE’s attention to forgetting and broader capability changes is useful here: the latest task score cannot describe the whole update. This does not require pretending to enumerate everything a model knows. It requires naming the commitments the organization has made and detecting when a later release stops satisfying them.

Recovery should be another ledger operation. Before promotion, I would identify a retained serving configuration and record which accepted changes reverting to it would discard. Restoring an older checkpoint may recover an earlier capability while reintroducing an obsolete answer. That trade needs explicit review, especially when urgent facts were supplied through retrieval while slower model updates were pending. Provenance helps operators reconstruct the available recovery choices; it does not itself reverse a learned association.

For an isolated research experiment with no downstream consumers and no promise that edits survive subsequent work, I would accept a versioned manifest and evaluation report instead of a dedicated ledger service. The boundary is repeated operational reliance. Once people depend on accumulated corrections across releases, the history must support queries about supersession, retention, and recovery — whether it lives in existing platform tables or a separate application.

I see this as data-platform work because identity, temporal validity, lineage, and durable evidence are already its responsibilities. Model teams should choose the update mechanism and design the evaluations; the platform should keep their claims inspectable across that mechanism’s lifetime. The useful release question is not simply whether the model learned the new answer. It is whether we can show what changed, what still holds, and what we would lose by going back.