Evaluation and observability harness for agent systems — a reference build

A reference build for the instrumentation an agent system needs before anyone should trust it: behavioral evals, trajectory replay, drift detection, and cost attribution, all projected from one typed run ledger.

Problem

Most agent evaluation stacks split the world in two. Production writes traces to one system; evals run through a second harness; dashboards aggregate whatever both paths happened to emit. That split is convenient, but it creates the exact failure a harness is supposed to prevent: the eval path and the production path drift apart, so a green eval can certify behavior adjacent to the thing that actually shipped.

The reference build rejects that split. The run ledger is the spine. A production run writes typed, append-only events. The production trace, replay fixture, eval trajectory, regression gate, drift window, cost report, and operational summary are all projections over that same event stream.

Constraints

This is a reference build rather than a vendor benchmark. The constraints are chosen to mirror production engineering seams: the trace has to be replayable without a second capture layer; eval results have to name the exact harness configuration that produced them; drift detection has to see segment regressions that aggregate metrics hide; and cost has to be reconstructible from step-level entries rather than trusted from a run-level counter.

Decisions

1. The ledger is the trajectory. The build follows the argument in agent observability means trajectory replay, not dashboards: eval fixtures are promoted production traces, not separately instrumented examples.
2. Evals are contracts. A suite is an executable gate over fixtures, judges, thresholds, and harness pins. This makes evals are the contract concrete: a score that cannot fail a gate governs nothing.
3. Drift is online behavior, not a stale benchmark. The harness includes synthetic production windows with step-change, slow degradation, and segment-local regressions, reflecting drift detection is the production eval.
4. Cost is reduced from the ledger. Per-step cost attribution is an event stream property. The run total is a reduction, not a mutable counter that can drift from its evidence.

Outcome

The working build is a public Python reference implementation with a toy support-triage agent, typed DuckDB-backed ledger, trajectory projection, replay fixture promotion, deterministic judge, thresholded eval suite, digest-enforcing regression gate, drift detector, cost reducer, and seven self-checking scenario scripts. The full source is public at github.com/Dhristhi/agent-eval-observability-harness. Running the verbose demo prints the production run, raw ledger events, trajectory projection, fixture promotion, eval contract, gate decision, drift alerts, and cost report from the same underlying run data.

The honest caveat: the build proves the architecture, not a calibrated production judge. The deterministic judge is intentionally simple, and a real deployment still needs human-labeled golden sets, judge calibration, and operational thresholds tuned against live traffic. The point of the build is the seam: no separate eval instrumentation, no unpinned scores, no dashboard-only observability.