Teach evaluation through disagreement, not answer keys

I would teach evaluation through disagreement, not answer keys. A canonical score can train learners to reproduce a verdict without understanding the measurement choices behind it. Disagreement forces those choices into view: what evidence counted, which criterion dominated, how uncertainty was handled, and whether two evaluators were even answering the same question.

Evaluation is a pipeline, not a lookup. The task must be specified, evidence collected, criteria interpreted, measurements produced, and a decision made under an error tolerance. Each stage can fail differently. A format validator may establish that JSON is parseable while saying nothing about whether its claims are supported. A functional test may prove that a tool was called while missing that the wrong authority approved it. A high average score can conceal one failure class that matters more than all the others.

I start exercises with paired artifacts that invite a defensible split. One agent reaches the right answer through an unsafe action; another stops safely with an incomplete result. One response is fluent but weakly grounded; another is awkward but cites decisive evidence. Learners score each independently, preserve their criterion-level reasons, and then investigate the spread. The aim is not to make every score converge. It is to locate which difference comes from evidence, which from rubric ambiguity, and which from a genuine value trade-off.

Hypothesis testing belongs in this lesson. A learner should state what a proposed metric is expected to detect, construct positive and negative cases, and use failure injection to test sensitivity. High sensitivity without specificity creates noisy gates; high specificity without sensitivity lets important failures through. An interval estimate is more honest than a point score when the sample is small. External verification matters when the evaluator shares assumptions with the system under test. I also ask learners to predict the disagreement before seeing results, then explain which assumption the observed spread contradicted.

This changes the role of the answer key. Instead of one approved number, I want an annotated decision record: admissible interpretations, disallowed shortcuts, decisive evidence, known uncertainty, and escalation conditions. Ground-truth annotation then becomes governed work rather than clerical labeling. The record can be revised when a new failure mode appears, and old results remain interpretable because the rubric version travels with them.

Preserving judge disagreement is especially important with model evaluators. Several calls to the same model family may repeat correlated assumptions, and majority voting can turn shared bias into apparent confidence. Learners should compare rationales before aggregation, test whether changing irrelevant details changes the verdict, and route high-variance cases to human review. The disagreement is diagnostic data.

There is one precise concession: answer keys remain effective for deterministic skills with stable semantics—syntax, arithmetic, protocol fields, or a known state transition. Inventing controversy there wastes attention. The boundary is whether correctness can be established by a fixed oracle. Once judgment depends on evidence quality, competing criteria, or uncertain consequences, the key should define a review process rather than pretend to eliminate interpretation.

The durable skill is not guessing what the grader wants. It is building and challenging a measurement argument. Learners should leave able to propose a fitness function, design a failure case, explain false positives and negatives, preserve dissent, and decide when evidence is insufficient. Evals are the contract, but contracts improve when their ambiguities are discovered before production. Disagreement is where that discovery begins.