Drift detection is the production eval — offline suites only catch what you predicted

An offline eval suite tests the failures you predicted, on the distribution you sampled when you built it. That's its definition, not its flaw. But production LLM systems degrade along axes nobody predicted — and against those, a frozen suite is structurally blind. So I've come to rank the eval investments in an order most teams invert: drift detection in production is the primary eval, and the offline suite is its supporting cast.

Count the ways a system that "passes evals" rots. Prompt distribution drift: your users change what they ask, how they phrase it, what language they ask it in — the suite still tests last quarter's traffic. Response quality and hallucination drift: slow degradation as context accumulates, retrieval corpora age, and usage wanders off the well-tested paths. And the one that still surprises teams: provider-side drift, where model behavior shifts behind an unchanged model ID because the provider updated serving infrastructure or quietly revised the model. Your code didn't change, your prompts didn't change, your suite is green — and your production behavior is different. No offline artifact can catch a change whose cause is upstream of everything you version.

The good news is that detecting this is not a research problem; it's statistical process control, a discipline manufacturing finished decades ago. Track rolling task-success and quality scores on control charts — Shewhart for step changes, CUSUM and EWMA for the slow drifts that individual-point alarms miss. Compare input distributions against baselines with KS tests and divergence measures. Calibrate window sizes and thresholds deliberately, because they encode the trade between sensitivity and alarm fatigue. And slice the analysis: aggregate metrics hide localized regressions, and "accuracy is flat overall" is compatible with "we broke everything for one customer segment."

What makes the statistics actionable is the deployment machinery around them. Sampled LLM-as-judge scoring gives you a continuous quality signal on live traffic. Shadow mode runs a new prompt or model invisibly against production input before it serves anyone. Canary rollouts step traffic up gradually with metric gates at each stage, and automated rollback triggers close the loop when a gate fails. Wire these together and the eval stops being a quarterly report and becomes a control system — detection, decision, and recovery on the same clock as the failure.

The concession is the ground-truth problem, and it's genuine: online monitoring mostly lacks labels. A judge model is itself a model — noisy, biased toward fluency, and capable of drifting exactly when the system it monitors drifts. Judge scores need periodic calibration against human labels, and no drift alarm tells you the answer is wrong the way a curated test case does. That's why the offline suite keeps its job: it is the pre-merge regression gate, the place where evals function as the contract for every prompt tweak and model upgrade. The two compose cleanly — offline gates what you predicted; online catches what you didn't.

But composition isn't symmetry. Teams routinely spend most of their evaluation effort curating offline test cases and treat production monitoring as a latency dashboard with a sentiment widget — nine parts prediction, one part observation, for a system whose defining property is that its inputs, its behavior, and its upstream dependencies all change without asking. Invert the budget. The suite you wrote is a hypothesis about how your system fails; production is the experiment that keeps running whether you watch it or not. Instrument the experiment.