Agent telemetry should preserve decisions, not merely token streams

Agent telemetry should preserve decisions, not merely token streams. I care less about reconstructing each token than about recovering the decision envelope: what the agent could have done, what evidence and policy constrained it, what it chose, and where that choice became consequential. A transcript shows the surviving path. Production diagnosis needs the alternatives that disappeared. Prompts, completions, and a final tool call capture behaviour, but not the mechanism engineers need to inspect, replay, or improve.

Tool selection makes the gap concrete. I want a structured event containing candidate tools, ranks, relevance scores, availability, expected latency, estimated cost, and the selection. Each rejected candidate needs a categorical reason—below threshold, unavailable, over budget, forbidden by policy, too slow, or missing a dependency—plus the constraint that fired. This is not a request to persist private chain-of-thought. It asks the harness to expose operational decision variables. “Database query rejected because estimated latency exceeded the five-second SLA” is queryable evidence; a rationale-shaped string is not.

The record also needs a point of commitment. Early steps precede everything that follows, so traces can make them look important merely because trajectories are ordered. I want telemetry to mark when an observation was consumed, an action crossed an approval boundary, or a mutation made the route costly to reverse. Attribution can then distinguish ancestry from leverage. An unused document deserves no causal credit; a tool result that changed the next action does. The useful grain is not “step three happened,” but “step three altered the reachable future here.”

OpenTelemetry supplies connective tissue, not decision semantics. I would propagate trace context across HTTP, IPC, queues, sidecars, and sub-agents, then activate the extracted context explicitly around asynchronous work so concurrent runs do not bleed together. Shared trace and span identities let candidate evaluation in one process join to policy checks, tool calls, and mutations elsewhere. The application must still emit decision events with stable schemas, iteration IDs, policy and model versions, and references to the state the agent actually saw.

With those records, replay becomes an intervention system rather than a video player. I can resample the model at one decision, substitute another action, alter a tool observation, change the supplied context, or swap the governing policy, then compare outcomes. Each intervention asks a different question: model variance, router quality, tool dependence, context sensitivity, or architecture choice. Token capture records only the realised branch; it does not preserve the variables needed to construct a credible counterfactual.

This changes how I design the observability store. Decision events are behavioural data with grain, keys, schema evolution, privacy boundaries, and replay dependencies. Candidate arrays increase volume, while tool IDs, user IDs, or free-form rejection reasons used as metric labels create ruinous cardinality. I keep low-cardinality aggregates in metrics, preserve structured decisions in traces or append-heavy event storage, and join them through stable run, step, session, tenant, and tool-call identifiers. The dashboard is a projection; the durable event model is the asset.

Retention should follow explanatory value rather than treating every field equally. I would keep long-lived aggregates for rates, latency, cost, and rejection categories; retain complete decision envelopes for failures, high-risk actions, policy denials, novel routes, and representative successes; and move older detail to cheaper storage when replay remains valuable. Sensitive prompts and tool results need redaction or shorter lifetimes, while compact facts such as candidate rank, policy version, commitment point, and mutation identifier can survive longer.

The bounded concession is that a deterministic, single-step workflow with no consequential tool choice may need only output, latency, error, and a sampled trace. If it always invokes the same read-only operation under a fixed policy, logging a synthetic candidate set adds ceremony without explanatory power. My claim begins where the runtime chooses among materially different actions, contexts, policies, or external effects. There, omitting rejected paths discards the evidence needed to explain the system.

I treat decision preservation as the contract between observability, replay, testing, and evaluation. Production traces should become fixtures; fixtures should admit controlled interventions; behavioural diffs should become regression cases; and eval gates should enforce permitted decisions, not merely permitted prose. A token stream tells me what the model said. A decision record tells me why this route remained available, why others closed, what committed the run, and which intervention would have produced a safer outcome. That is telemetry an agent platform can learn from.