Structured-output evals must test recovery, not just valid JSON

I think structured-output evals must test recovery, not just valid JSON. Syntax compliance proves that a parser can read the response. Production reliability depends on whether the values are correct, failures are classified, repair is bounded, retries preserve intent, and the pipeline reaches an honest terminal state when no safe output can be produced.

The evaluation stack has three necessary tiers. Structural checks ask whether the output is complete JSON. Schema checks enforce types, required fields, enums, unions, bounds, and whether undeclared fields are rejected. Semantic checks ask whether the values actually represent the source and task. Constrained decoding can make the first two tiers nearly perfect while still returning a beautifully typed falsehood.

That is why golden records need field-level meaning, not only serialized examples. A useful case identifies which values are deterministic, which allow a set or range, which require a calibrated judge, and which demand human review. Adversarial cases should include ambiguous input, contradictory evidence, missing context, oversized values, malicious instructions, optional-field traps, and schema versions that expose brittle assumptions.

Recovery deserves its own assertions. When the first response fails, did the system distinguish truncation from type mismatch, missing evidence, semantic contradiction, provider refusal, and transport failure? Did it retry only failures that could change? Did the reask describe the defect without leaking untrusted output into a more privileged prompt? Did repair preserve the intended values instead of merely coercing them into acceptable types? Each recovery path should preserve the original request and its evidence.

I would measure first-pass structural, schema, and semantic success separately from eventual success. Then I would add repair success by failure class, retries per successful output, tokens and latency per success, false-repair rate, truncation rate, and terminal abstention quality. An eventual-valid rate can hide a pipeline that burns three calls on every request or silently changes a customer identifier from a string to an integer.

Token limits are part of the test surface. A schema with nested arrays may pass small examples and truncate on realistic records. Size the output budget from the observed distribution of representative cases, then test beyond the high percentile and verify that truncation is detected as truncation. A partial object should not enter generic JSON repair and emerge as apparently complete business data.

Schema evolution makes regression testing unavoidable. Run the same task across old and new schemas, prompts, and models, and make any re-baseline an explicit review decision. A new optional field can alter generation enough to degrade existing values. As structured outputs are an integration contract, compatibility includes behavioral meaning, not just whether the new parser accepts the old bytes.

I concede one precise boundary: for a deterministic formatter wrapping trusted, already-validated values, structural and schema checks may cover nearly all meaningful risk. Semantic recovery testing earns its cost when the model extracts, classifies, infers, or chooses values from ambiguous evidence—the common case for LLM pipelines.

A passing eval should therefore tell me what happened after failure, not merely that most first attempts parsed. Tool errors must be typed for the agent, and structured-output failures need the same discipline. The reliable outcome may be repaired data, a bounded retry, an escalation, or an explicit refusal. Testing those paths makes the contract honest when valid JSON is not the same thing as a valid result.