Context caches need invalidation contracts, not hit-rate targets

I think context caches need invalidation contracts, not hit-rate targets. In an agent system, a cache hit is not automatically a success. It may return an old policy, a superseded tool schema, stale retrieved evidence, the wrong tenant’s session projection, or a summary built before a critical correction. Faster access to invalid context makes the system wrong more efficiently.

The useful mental model is that context is a cache of authoritative state, not the state itself. Long-running harnesses already benefit when progress, decisions, and intermediate results live in files or databases and only the relevant subset enters the model window. Once context is treated as a materialized projection, the cache question becomes precise: which source versions produced this projection, for whom, under which policy, and until what event remains it valid?

I would put those answers in the cache identity. A useful key includes more than prompt text. It may need tenant and user scope, conversation or task identity, model and tokenizer version, system-prompt release, tool-schema digest, policy version, memory or retrieval index version, and the freshness class of referenced evidence. Omitting one of these dimensions can turn reuse into behavioral cross-contamination. As cache keys are behavior contracts, two requests should share an entry only when the system is willing to treat their relevant behavior as equivalent.

Time-to-live is only the weakest form of invalidation. Some context can tolerate age; other context must change immediately when its source changes. A customer’s preferred tone may survive for weeks, while revoked authority cannot survive another tool call. A summarized conversation should be invalidated when a correction changes its meaning. A retrieved compliance answer should carry the effective and verified time of its evidence, not merely the time it entered the cache.

This suggests an event-driven contract. Policy publication invalidates projections tied to the previous policy version. A tool-schema change invalidates cached plans that mention the old contract. Memory updates invalidate affected user or entity projections. An index promotion changes the eligible retrieval snapshot. Permission revocation bypasses reuse entirely. The invalidation event, affected scope, maximum propagation delay, and fallback behavior should be owned before a hit-rate dashboard is celebrated.

Versioning makes incidents recoverable. If every cached projection records its source versions, an operator can explain why a model saw a particular context and invalidate only the affected entries. Without that lineage, the safest response to a context defect is a global flush—and even that may fail if another layer silently repopulates from the same stale source. Context assembly is schema design; cached context therefore needs schema and source lineage like any other derived data product.

Metrics should follow the contract. I care about stale-hit rate, cross-scope rejection, invalidation propagation time, rebuild latency, bypass rate for high-consequence queries, and behavior differences between cached and freshly assembled context. Hit rate and token savings remain useful, but only after correctness. A cache that saves half the tokens while serving obsolete policy is not an optimization; it is an unmeasured release path.

I concede one precise limit: immutable prompt prefixes and static reference material can often use simple content-addressed caching without a rich event system. If the bytes and every behavioral dependency are truly immutable, their hash is the invalidation contract. The mistake is extending that simplicity to projections whose meaning depends on mutable state, policy, identity, or time.

A context cache should be allowed to miss safely. When freshness cannot be proven, the harness can rebuild from authoritative state, narrow the answer, ask for confirmation, or abstain. That is especially important because knowledge staleness is a lifecycle problem, not a ranking defect. I would accept a lower hit rate in exchange for explicit equivalence, traceable lineage, bounded staleness, and a known fallback. Those are the properties that make cached context operationally trustworthy.