Agent observability should record decision deltas, not just spans

Agent observability should record decision deltas, not just spans. A span can tell me that the model ran for 1.8 seconds and called a tool. It rarely tells me what changed between “search again” and “send the answer,” which evidence caused that change, or why a candidate action disappeared. Those differences are where agent behavior becomes diagnosable.

Conventional tracing is organized around execution boundaries: service, operation, parent, duration, status. That remains necessary. Agent failures, however, often occur while every span is green. A model updates an assumption from stale evidence, drops a constraint during compaction, switches tools after a retry, or accepts a lower-confidence plan. The infrastructure completed correctly; the decision state moved incorrectly.

I model each consequential step as a before-and-after record. The delta includes changed goals, beliefs, constraints, evidence references, candidate tools or plans, policy verdicts, confidence, unresolved errors, and intended effects. It does not need a full hidden chain of thought. Structured reasons such as “candidate rejected: missing tenant scope” or “plan changed: inventory check timed out” are more stable, queryable, and governable than unrestricted reasoning text.

This sharpens the claim that agent observability means trajectory replay. Replay requires the state that informed each decision, but storing complete snapshots at every turn is expensive and makes comparison difficult. A versioned base state plus ordered deltas can reconstruct the trajectory while exposing the exact transition. It also creates a natural place to attach model version, prompt digest, retrieved artifact IDs, tool schema, policy revision, and execution identity.

Rejected alternatives matter because incidents often hide in the road not taken. If an agent considered the safe tool, rejected it after a malformed response, and selected a broader fallback, the final tool-call span preserves only the fallback. Debugging must preserve rejected alternatives so an operator can distinguish bad selection from missing capability, bad evidence, and an overly permissive recovery rule.

Decision deltas also improve evaluation. Instead of scoring only the final answer, an eval can assert that a policy denial narrowed the candidate set, that new evidence revised a belief, or that lower confidence caused escalation. Evals are the contract; deltas expose intermediate contractual states without demanding deterministic prose. Drift detection can then compare distributions of tool-selection changes, escalation transitions, or evidence substitutions across releases.

The data model needs restraint. High-cardinality artifacts belong in governed storage with hashes and retention rules, while the trace carries bounded identifiers and typed changes. Sensitive prompts, personal data, and unrestricted reasoning should not become permanent telemetry by default. Telemetry should preserve decisions, but preservation still requires minimization, access control, and deletion semantics.

There is one precise concession: ordinary spans and aggregate metrics are enough for deterministic, stateless model calls where the application makes every consequential decision in code. In that boundary, latency, errors, tokens, and input-output sampling may answer the operational question. Once the model selects actions, revises state, or controls recovery, span-only visibility becomes an incomplete behavioral record.

I want an incident query to answer: what changed, because of which evidence, under which policy, and with what resulting effect? That query should work across model, tool, and human handoffs, because a decision boundary does not stop being consequential when it crosses a service boundary. A trace waterfall can show where time went. A decision-delta ledger shows how the system became the system that acted. Production observability needs both.