Semantic caching is the only cache that can lie to you
Every cache I have ever operated shared one saving grace: it could be slow, it could be cold, it could be full, but it could not be wrong. Hit means exactly what was stored for exactly that key; miss means do the work. A semantic cache breaks that contract — it is the only cache in the stack that can return a confident, fluent, incorrect answer, and it does so at cache speed with cache authority.
The mechanism makes the failure structural, not incidental. A semantic cache — GPTCache is the reference architecture — embeds the incoming query, runs nearest-neighbor search over stored query embeddings, and returns the cached response if similarity clears a threshold. The "key" is not a key; it's a region in embedding space. Two queries can sit at 0.9 cosine similarity and demand different answers — same question about a different contract, a different quarter, a different patient. Cross the threshold boundary in the wrong direction and the user receives an answer to a question they didn't ask, styled as if the system did the work. The literature calls this a false positive, and it's the defining trade: calibration guidance runs 0.85–0.90 for factual lookups and 0.92–0.95 for open-ended generation, which is another way of saying the correct threshold is workload-specific and there is no safe default. A miss costs one LLM call — 500 milliseconds to 5 seconds. A false positive costs a wrong answer you will probably never detect, because nothing errored.
The contrast with provider prompt caching sharpens the point. Anthropic- and OpenAI-style prefix caching is byte-exact matching on the token prefix: deterministic, roughly 90% cheaper on hits, and incapable of lying, because equality is checked, not estimated. The two layers are complementary, but only one of them is actually a cache. The semantic layer is a retrieval system wearing a cache's costume — embedding model, ANN index, similarity evaluator, relevance decision. That's the same machinery as your RAG stack, and it inherits the same obligations: you wouldn't ship a retriever without measuring precision, yet teams routinely ship semantic caches monitoring hit rate alone, which counts the money saved and never the answers corrupted.
Operating it honestly means engineering the correctness back in. Compound cache keys that fingerprint the system prompt, conversation context, and tool state, so a hit means "same question in the same situation." Per-user scoping, because a cross-user false positive is a privacy incident, not just a wrong answer. Temperature bypass, so stochastic outputs aren't reused as if they were canonical. Invalidation treated as data lifecycle — TTLs, event-triggered purges when the underlying facts change, version tags tied to prompt and model releases. And post-retrieval validation on the hit path, with false-positive rate tracked as a first-class health metric next to hit rate.
The concession is that the economics are genuinely compelling, which is why the risk gets rationalized. A cache hit answers in 10–100 milliseconds against the LLM's seconds; a 30% hit rate can roughly halve effective latency and take a proportionate bite out of spend, and for high-volume repetitive workloads — support queues, FAQ surfaces — that math is hard to walk away from. The conclusion isn't to avoid semantic caching. It's to stop procuring it as infrastructure and start operating it as a model: thresholds are tunable parameters, the threshold decision is an inference, and inferences need evals.
My rule: a semantic cache goes into production with the same ceremony as a new retriever — a labeled test set, a measured false-positive rate at the chosen threshold, and an owner for both numbers. If that sounds like too much work for a cache, that's the point. It isn't one.