Agent reliability is failure-mode design, not uptime
The reliability vocabulary we inherited from infrastructure — uptime, error rates, nines — assumes that failure announces itself. An agent can be 100% "up" and failing constantly. Every API call succeeds, every tool returns, latency is green, and the agent is calling the wrong tool with plausible arguments, silently skipping a verification step, or compounding one small early mistake into a confidently wrong final action. Availability measures none of that.
Distributed-systems engineers call the well-behaved kind of failure fail-stop: a component either works or visibly halts. Almost nothing an agent does wrong is fail-stop. When I catalogue what actually breaks in agent systems, the list reads nothing like an infrastructure runbook. Semantic tool errors, where the call is syntactically perfect but the tool is wrong or the arguments are logically invalid. Plan hallucination and plan rigidity. Tool thrashing — many calls, no progress. Doom loops and retry storms. In multi-agent setups it gets worse: hallucinations cascade across handoffs, agents develop split-brain views of shared state, and trust-boundary confusion lets one agent's bad output become another's ground truth. Every one of these presents as success at the transport layer. Your dashboard shows a healthy system while the work product rots.
From "it works" to invariants
This is why I've come to treat agent reliability engineering as a discipline of enumeration, not monitoring. The framework I keep returning to replaces the vague "it works" criterion with three formal invariants. A liveness invariant: every task eventually completes or fails explicitly — no infinite loops, no deadlocks, no silent stalls. A safety invariant: no harmful state transition on any execution path, which must hold absolutely, not probabilistically. And economic boundedness: cost stays predictable across all trajectories, because an agent that succeeds while burning unbounded tokens has still failed. None of these are enforced by better prompting. They're enforced by architecture — watchdog timers, circuit breakers on tool calls, checkpoint and rollback, explicit fault boundaries that act like bulkheads at the seams where skills compose.
Measurement has to move to the same layer. The service-level objectives that tell you something useful about an agent are semantic ones: chain completion rate (the share of multi-skill chains finishing without human fallback — realistic targets sit around 85–92%), routing accuracy (90–96% of skill selections invoking the intended skill), and inter-skill error rate (under 3% of handoffs propagating errors downstream). When one of these degrades, you can localise whether the failure is in a component or in the composition — which uptime, by construction, cannot tell you.
Enumerate, then design
The practical discipline is a failure-mode inventory before launch: classify candidate failures along temporal, spatial, and severity axes, then map each mode to a mitigation. Retry-induced duplication gets idempotency keys. Retry storms get capped exponential backoff with jitter and a single-layer retry discipline. Flaky tools get three-state circuit breakers. Runaway sub-agents get bulkheads and timeout hierarchies. The mitigation catalogue is mostly borrowed from twenty years of distributed systems — what's new is that the failures being mitigated are semantic, so detection needs semantic instruments too.
The honest caveat: uptime is not irrelevant — an agent sitting on a down model API is failing by any definition, and infrastructure hygiene is still table stakes. And no enumeration is ever complete; some failure modes only reveal themselves under real traffic, which is why fault injection and chaos experiments belong in the programme rather than being treated as paranoia. For a low-stakes agent with a human reviewing every output, full invariant engineering is genuinely overkill. But the moment an agent acts autonomously on systems that matter, the reliability question changes shape — from "is it up?" to "what are all the ways it can be wrong while up, and what structure catches each one?" That second question is the job. It's also why so many impressive demos never survive contact with production — the gap is architectural, not cosmetic.