Health checks should measure recoverable user outcomes

Health checks should measure recoverable user outcomes, not merely confirm that a process answers on a port. A green endpoint can coexist with stale indexes, exhausted queues, broken credentials, unavailable dependencies, or work that completes but cannot be recovered after interruption. “The container is alive” is useful infrastructure evidence. It is not yet evidence that the service is healthy.

I separate liveness, readiness, and outcome health because they answer different questions. Liveness asks whether restarting the process may help. Readiness asks whether this instance should receive new work. Outcome health asks whether a representative user intention can traverse the system and reach an acceptable, recoverable state. Collapsing all three into one endpoint creates bad automation: deep dependency checks can trigger restart storms, while shallow checks keep irreparably degraded instances in rotation.

The outcome must be concrete. For a retrieval service, health is not just an HTTP response from the query API; it includes index freshness, evidence availability, and a bounded fallback when the preferred index is stale. For an agent runtime, it includes accepting work, recording the decision state, executing or refusing effects under policy, reaching a terminal state, and resuming from the latest valid checkpoint. For a data product, it includes freshness and integrity at the output a consumer actually uses.

Recoverability changes the metric. A request that succeeds once but loses its state on restart is not healthy for durable work. A queued task with a rising age may be more urgent than low CPU utilization. A dependency timeout may be tolerable if the system degrades honestly and preserves a retryable state; a fast partial response may be worse if it commits an irreversible effect without evidence. I want health indicators tied to the next safe action—continue, drain, retry, fail over, or escalate.

This means external monitoring must complement internal telemetry. Internal metrics describe what components believe about themselves. Synthetic transactions exercise the boundary users cross. Flow monitoring correlates stages instead of declaring each service independently green. Historical baselines reveal a freshness or latency change that remains inside a static threshold. And the observability stack is a data platform because these signals need shared identity, retention, lineage, and queryable context during diagnosis.

Incident management should consume the same model. An alert that says “health check failed” has discarded the useful information. The incident trigger should name the affected outcome, last known good state, failed stage, recovery option, and confidence in the diagnosis. Severity then follows consequence and recoverability, not the number of red widgets. The platform should also monitor the monitoring system—missing probes and delayed telemetry are health failures of the evidence path itself.

There is one precise concession: a shallow process check is sufficient for a stateless, disposable component when upstream systems independently verify the user outcome and retries are safe. Deep checks inside every replica would add load and couple restart behavior to remote dependencies. The boundary is ownership: if another control actually proves and recovers the outcome, the local check can stay narrow. If nobody does, the simplicity is only an observability gap.

I would build a health hierarchy from cheap local facts to periodic end-to-end probes, then test the recovery transitions those probes are meant to trigger. Reliability is failure-mode design, so a health signal earns its place only when it distinguishes a meaningful failure and leads to a safe response. Green should mean more than breathing. It should mean a user outcome can complete—or, when it cannot, the system can preserve truth, contain effects, and recover deliberately.