Streaming storage should be selected by replay and compaction economics, not latency alone

Streaming storage should be selected by replay and compaction economics, not latency alone. The live path gets the benchmark charts, but the retained path determines what the system costs to recover, reprocess, audit, and keep useful. A platform that delivers an event in two milliseconds and makes six months of history ruinously expensive has optimized the smallest part of the lifecycle.

I start with the question the stream will have to answer after today. Can a new consumer replay from the beginning? Can an operator rebuild state after a bad deployment? Can an agent reconstruct why it acted from the events available at the time? Event sourcing earns its complexity only when replay is the feature; once replay is the feature, retention layout and read economics become first-order architecture.

The storage substrate is changing around that requirement. Diskless Kafka designs put durable segments in object storage and leave protocol servers largely stateless. Flink separates state from compute. Streamhouse designs expose one dataset as a log, a key-value view, and a table. The attraction is not merely cheaper disks. It is the possibility of storing an event once and reading it through different operational and analytical shapes without maintaining a permanently duplicated speed layer.

That promise has a price curve. Object storage makes long retention cheap, but every round trip has a latency floor. Group commit amortizes writes by buffering them, which converts cost into delay. Local NVMe caches make warm replay fast, while cold replay still pays for remote objects and index reconstruction. The right question is therefore not “How fast is the broker?” but “What combination of hot window, cold history, recovery-time objective, and reread frequency produces an acceptable bill?”

Compaction belongs in the same calculation. A key-compacted stream can preserve the latest value per subject while discarding superseded state; a time-retained log preserves the transitions needed for audit and reconstruction. JetStream exposes those choices directly through limits, interest, and work-queue retention, per-subject caps, roll-ups, and sealed streams. None is universally correct. A task queue wants deletion after acknowledged work. An agent decision ledger wants immutable history. A configuration stream may want latest-value semantics plus revisions.

Recovery turns configuration into economics. Keeping three replicas locally can reduce failover time but multiplies cross-zone writes and attached-disk cost. Object-storage-native logs can rebuild brokers far faster because durable bytes do not move with the failed node, but the recovery path now depends on metadata integrity, cache warming, and remote-store throughput. The cheapest steady state can be the most expensive outage if nobody priced the cold path.

This is also why streaming as the default temperature does not mean every byte deserves equal heat. Consumer lag, replay frequency, legal retention, compaction ratio, and restore objectives should place data into tiers. The protocol can remain Kafka- or NATS-compatible while the storage and retention policy varies by topic. “One streaming platform” should not mean one durability setting.

There is one precise concession: for an ephemeral control signal with no audit value, no late consumers, and a producer able to recreate state, live-path latency can dominate the decision. NATS Core’s at-most-once pub/sub is sensible for token fan-out or transient presence. The stronger lifecycle analysis begins when losing history changes correctness, recovery, compliance, or the cost of onboarding a new consumer.

I would choose streaming storage from a worksheet containing hot-path latency, retained bytes, compaction semantics, expected replays, cold-read throughput, failure-domain replication, and recovery-time objectives. The lakehouse is a natural substrate for durable agent data because cheap history and multiple read shapes matter. Streaming platforms should be judged by the same standard: not only how quickly they move the present, but how economically they preserve and reconstruct the past.