Grade calibration and escalation, not fluent answers

We should grade agents on whether they know when not to answer. Fluency makes an unreliable system look finished: the response is grammatical, complete, and confident even when the underlying knowledge is weak or the task has crossed an authority boundary. An evaluation that rewards only answer quality trains the wrong behavior. I want to measure whether the agent chooses correctly among answering, retrieving evidence, asking for clarification, spending more reasoning budget, or escalating to a human.

The metacognition literature gives this a useful structure. Nelson and Narens separate object-level cognition from a meta level that monitors and controls it. Applied to an agent, generating an answer is object-level work; recognizing uncertainty and changing strategy is meta-level work. Current language models are strong at plausible completion and much weaker at the internal signals that should interrupt completion. They can reflect on a past failure, as Reflexion-style loops do, but they lack a dependable equivalent of the human conflict signal that says, before commitment, “this path is not trustworthy.”

That gap means calibration cannot be inferred from tone. A model can say “I’m uncertain” mechanically and still be badly calibrated; it can sound assured while operating outside its reliable knowledge. Evaluation has to compare expressed confidence and chosen action with observed correctness across many cases. Did the agent reserve high confidence for answers that were actually right? Did it retrieve when evidence was available? Did it ask a question when ambiguity changed the answer? Did it escalate when the required judgment exceeded its mandate? The score is not eloquence. It is the quality of the control policy around eloquence.

Benchmark design often hides this problem by requiring a completion for every task. Pass rates and functional correctness matter, but they reward attempts even when abstention would be safer in production. I would add cases with missing information, conflicting evidence, stale context, untrusted tools, and actions above the agent’s authority. A good agent should fail some of these by design: refuse the action, request the missing fact, or hand the case over with a useful summary. Recovery error rate and trajectory grading then matter alongside final-answer accuracy because the route through uncertainty is part of the result.

This is also a better teaching objective for people who supervise agents. Prompt tricks age quickly; judgment about evidence, scope, and escalation compounds. A learner should be able to inspect an agent run and distinguish a knowledge gap from a tool failure, a low-confidence inference from a policy boundary, and a recoverable error from a reason to stop. That skill supports calibrated trust rather than reflexive acceptance or reflexive skepticism.

There is one narrow concession. In deterministic, closed-book tasks with complete inputs and an execution-based judge, final correctness may be sufficient because uncertainty and authority have been designed out of the exercise. But most valuable agent work is open-world, tool-mediated, and consequential. There, forcing every case into an answer is not rigor; it is a benchmark artifact.

I would therefore put an escalation matrix beside every answer rubric. For each test, define what the agent could know, what evidence it could acquire, what it was allowed to do, and which uncertainty should trigger a different path. Then score both the outcome and the choice of path. The agent that gives fewer fluent answers but makes better decisions about when to proceed is the one I would trust in production — and the one our training should reward.