Shadow evals should precede every model promotion

I think shadow evals should precede every model promotion. Offline suites tell me whether a candidate still handles cases I anticipated. Shadow execution tells me how it behaves on the current mix of real requests, context, tools, languages, edge cases, and operating constraints—without letting the candidate’s answer or action reach the user.

A model change is a behavior release even when application code is untouched. The new model may follow instructions differently, choose another tool, produce a subtly different schema, consume more reasoning tokens, refuse different requests, or shift latency at the tail. Aggregate benchmark gains do not bound these changes. A candidate that is better on average can still regress the narrow workflow that pays for the system.

Shadowing creates the missing comparison. The production version handles the request normally while the candidate receives a safe copy of the same eligible input. Its outputs, tool intentions, trajectory, latency, token use, and policy decisions are recorded but effects are suppressed or simulated. The comparison is paired at the request level, so traffic mix cannot explain away the delta. It also reveals consequential disagreements that a static expected answer would never represent.

The promotion gate should be versioned before the run begins. I want deterministic checks for schemas and invariants, pairwise quality judgments, trajectory checks for tool sequence and prohibited actions, cost and latency ceilings, and explicit non-inferiority margins. Results should be stratified by task type, customer segment, language, tool domain, and risk class because an aggregate score can hide a severe local regression.

Representative traffic is the advantage and the risk. Production prompts may contain personal data, secrets, or regulated content that a new provider is not yet allowed to receive. Shadowing therefore needs eligibility policy, minimization, de-identification where faithful, retention limits, and a guarantee that shadow tool calls cannot commit effects. Agent privacy is a data-flow problem; an eval label does not exempt the shadow path from that architecture.

The comparison must also preserve uncertainty. LLM judges can prefer style over correctness and may share blind spots with the candidate. I would combine deterministic assertions, calibrated judges, human review of disagreements, and targeted replay for high-consequence trajectories. That is consistent with evals as an executable contract: no single metric owns the promotion decision.

Shadow results should end in an explicit verdict: promote, promote for a bounded segment, revise and repeat, or reject. The record needs model identity, prompt and policy releases, dataset window, eligible traffic, excluded classes, gate definitions, score distributions, disagreements, cost, and the approver. Without that release record, shadow mode becomes an impressive dashboard rather than a control.

I concede one precise boundary: a zero-retention system may be unable to replay or duplicate sensitive production inputs, and some tool environments cannot be simulated faithfully. In that case, a de-identified representative corpus plus a tightly bounded canary may replace full shadowing. The obligation to compare behavior before broad promotion remains; only the evidence path changes.

Offline regression tests, shadow evaluation, and post-release monitoring form a sequence, not competing methods. Offline tests catch known failures cheaply. Shadowing exposes distribution and integration changes before users carry the risk. Production drift detection watches what remains after release. Skipping the middle step turns every model upgrade into an uncontrolled experiment on the people and workflows the system is meant to serve.