Your observability stack is a data platform wearing a dashboard

Your observability stack is a data platform wearing a dashboard. The dashboard is what executives and on-call engineers see, so it gets mistaken for the product. But the product is the event substrate underneath it: trace schemas, ingestion contracts, identity keys, retention rules, cost records, eval scores, lineage, and the query engine that decides whether a production incident can be explained before everyone forgets the shape of the failure.

This matters more for agents than it did for services because the unit of explanation is richer. A web service can often be understood through latency, error rate, and a few structured logs. An agent run needs prompts, retrieved context, tool calls, tool results, routing decisions, model versions, token accounting, evaluator scores, human annotations, and sometimes the exact boundary conditions of a nondeterministic run. OpenTelemetry's GenAI events and the Langfuse/LangSmith/Phoenix style of trace objects are useful not because they add prettier dashboards, but because they normalize this mess into records that can be joined. The moment an eval score is attached to a trace, observability stops being monitoring and becomes a behavioral dataset.

That dataset has all the old data-platform problems, just under operational pressure. You need stable run IDs and session IDs so one decision trajectory can be reconstructed. You need schema evolution because today's trace payload will not contain tomorrow's evaluator fields. You need tenant and user keys for cost attribution and privacy boundaries. You need columnar storage and retention classes because raw prompts are hot for debugging and dangerous to keep forever. ClickHouse is popular in this space for the same reason it works for audit and event analytics: append-heavy ingestion, sparse indexes, compression, low-cardinality dimensions, TTLs, and sub-second slice-and-dice over immutable streams are exactly the shape of the workload. Whether the engine is ClickHouse or something else, the architectural category is not “dashboard.” It is OLAP over behavioral event streams.

The practical failure mode is buying observability as a screenshot generator. Teams collect traces but cannot answer which model route increased cost per successful task, which customer segment started failing after a prompt change, whether provider drift occurred behind an unchanged model ID, or which eval score regressed for one tool path while the aggregate stayed flat. Those are data questions. They require grain, keys, history, and query shape. They also require governance: PII redaction before long-term storage, column-level TTLs for sensitive fields, and audit trails for who inspected what. At that point you are doing data platform engineering, even if the UI says observability.

The concession is that not every agent system deserves a warehouse-grade telemetry stack on day one. A small internal assistant can live with sampled traces, a hosted tool, and a handful of dashboards. Overbuilding observability is real; it turns every prototype into a platform migration. But the decision should be made from the workload, not the label. If the agent takes consequential actions, crosses tenants, uses multiple models, or must survive audits and regressions, the data platform is not optional. You can rent the UI, but you still have to own the grain of the events.

This is why I like the phrase “wearing a dashboard.” It keeps the useful part of the dashboard in its place. Visuals are projections. Alerts are projections. Eval comparisons are projections. Cost reports are projections. The thing they project from is the durable event model. Design that first, and observability becomes the nervous system of the agent platform. Design only the dashboard, and you get a screen that glows while the system forgets what happened.