Agent observability means trajectory replay, not dashboards
A dashboard can tell you that 4% of agent runs failed yesterday. It cannot tell you why run 4173 refunded the wrong customer. Those are different questions, and only the first is answered by metrics. The unit of observability for an agent is the complete decision trace — every prompt, every tool call and its result, every routing decision, and the state the agent held at each point — and the ability to replay that trace should be a design requirement, not a forensic afterthought.
The classic three pillars — logs, metrics, traces — carry over from services, but the weight shifts decisively onto traces, and the trace itself changes shape. An agent run is not a linear request path; it's an execution graph, often a DAG with parallel sub-agents, where causality runs through span links rather than tidy parent-child nesting. The OpenTelemetry GenAI semantic conventions now give this a standard vocabulary: a root span for the full agent invocation, per-iteration step spans carrying step number and type, tool-execution spans with tool name and call ID, a conversation ID to correlate multi-turn sessions, even per-step cost recorded as span events and aggregated into a cost counter by model and session. If you adopt one thing from this note, adopt that vocabulary early — retrofitting semantic conventions onto a bespoke logging scheme is miserable work.
Capture is necessary; replay is the point
Why insist on replay rather than just rich traces? Because agent debugging is causal, not statistical. The visible error is rarely the root cause — a wrong final action usually traces back through a chain of individually plausible decisions to one early misreading, and reconstructing that causality chain requires stepping through the trajectory as the agent experienced it. That imposes concrete capture obligations that ordinary tracing skips: snapshot the assembled context window at each decision point, not just the final prompt template; assign every run a globally unique run ID that keys the whole record; and store the non-deterministic inputs — retrieved documents, tool responses, sampled outputs — because the model itself won't give you the same answer twice. Perfect determinism is unachievable, but approximate replay from stored inputs is enough to answer "what did it know, and when?"
Replay also stops being purely retrospective once you have it. LangGraph Studio's checkpoint-based node replay lets you re-run execution from any node; counterfactual router replay lets you re-route from a past state snapshot and ask "what would have happened had it picked the other tool?" That's not observability as autopsy — it's observability as experiment. The same captured trajectories then feed evaluation: yesterday's production failure, replayed, is today's regression case, which is how the eval contract grows teeth. And none of it works unless the trajectory was durably persisted in the first place — replay and checkpointed execution are the same investment viewed from two sides.
The honest cost
Full trajectory capture is not free, and here's the concession: at serious volume you will not keep everything. Context snapshots are bulky, prompts and tool results carry PII that needs redaction before storage, and doubling your storage bill to debug a reporting agent nobody audits is a poor trade. The standard resolution is tail-based sampling — decide what to keep after the run completes, keeping 100% of error trajectories and slow outliers while sampling normal traffic at a low rate. For a simple single-model pipeline with static prompts, metrics plus sampled logs genuinely suffice. But the moment an agent makes autonomous multi-step decisions against real systems, the calculus flips: the first production incident you cannot explain will cost more than a year of trace storage. Dashboards tell you that you have a problem. Trajectories tell you what to fix. Build for the second question from day one, because you cannot retroactively capture the trace of a failure that already happened.