A typed confidence field still needs a local calibration contract

I want a typed confidence field to carry a local calibration contract before it carries operational authority. Jev makes uncertain judgments easier to integrate, but a clean response type cannot establish what its numbers mean on my workload. My contract would name the predicted event, the population, the model and question versions, the evidence supporting the threshold, and the fallback when that evidence no longer applies. Without those boundaries, confidence becomes an attractive field name attached to an undocumented decision policy.

The distinction starts with the primitives. TypeSafe documents Choice as selecting the highest-probability option from a supplied set, while returning the full distribution. Score returns a probability-weighted position across ordered rubric levels, alongside their probabilities. Noul returns the probability that a yes/no proposition is true. A middle Noul means uncertainty between yes and no, not moderate intensity. It has no separate confidence field. A Boolean produced by thresholding it belongs to application policy—not a fourth documented wire primitive.

For Choice and Score, TypeSafe explicitly describes confidence as a statistic derived from the distribution. It summarizes concentration; it is not another observation of correctness. I would therefore preserve the selected probability, the full distribution, and the vendor confidence separately. A Score mean can conceal disagreement between distant levels. A decisive Choice can still choose among options that omit the right answer. Neither problem disappears because the response passes validation. This is where the integration contract needs an empirical companion.

Calibration asks a different question: across comparable predictions assigned a given probability, how often does the defined event occur? The Vidhyarthi calibration evidence and Jev synthesis preserve that population-level distinction. They also separate TypeSafe’s stated RLCD training objective from independent deployment evidence. I would not translate Reinforcement Learning for Calibrated Decisions into a local correctness guarantee. The reviewed research contains no live Jev experiment, and the current primitive documentation supplies interfaces and vendor examples—not my application’s measured reliability.

My first contract would be deliberately narrow: whether a support ticket belongs to the selected team under a frozen routing taxonomy. It would not silently expand into whether that team resolves the issue, whether a refund is warranted, or whether payment is authorized. I would retain question wording, option descriptions, model identity, input provenance, and adjudicated labels with every evaluation record. Changing the taxonomy or rubric would trigger review of the calibration evidence, even if the JSON schema stayed identical.

I would then compare predicted event probabilities with observed outcomes on representative held-out tickets, including rare routes and cases where none of the options fits. Reliability diagrams would show the relationship; bin counts and uncertainty intervals would show how little evidence some regions contain. I would report probability loss and task accuracy separately, then examine error among accepted decisions against automation coverage and review volume. If I fitted a post-hoc mapping, its fitting data would stay separate from the final evaluation. These are proposed tests—I have not run them.

Thresholds would follow action costs, not naming conventions. A mistaken queue assignment and a mistaken refund approval cannot inherit the same operating point merely because both expose a number between zero and one. I would sample reviewed and automatically accepted cases, inspect language and route slices, and recheck after model or question changes. Calibration cannot repair missing evidence, and even a well-calibrated prediction cannot supply consent. The fallback and authorization rules remain deterministic application responsibilities.

I would accept raw confidence as a ranking signal for reversible, low-stakes triage if held-out testing demonstrated an acceptable error–coverage trade-off; I would not require a probability interpretation that the application never uses. That is the precise limit of my demand. Once a team presents the value as a likelihood of correctness, or uses it to justify consequential automation, it owes a stronger contract. I want typed uncertainty because it makes that contract inspectable—not because it lets me avoid establishing it.