The standard objection to evaluation-first delivery arrives on schedule in every engagement: our work is judgment, not lookup. There is no answer key. You cannot grade taste.
The objection is half right. Many enterprise decisions have no single correct answer. But they do have a distribution of answers your best people would accept, and that distribution is measurable. The question is never whether the agent is right in some platonic sense. It is whether the agent decides the way your strongest operators decide, at their consistency or better.
Rubrics beat answer keys
For judgment work, the golden set carries rubrics instead of answers: the factors a senior underwriter weighs, the thresholds a claims lead applies, the red lines nobody crosses. Graders score the agent's reasoning against the rubric, not its output against a key. Agreement between human graders becomes the ceiling; the agent's job is to approach it.
You are not measuring truth. You are measuring fidelity to your best people's judgment.
The disagreement dividend
Building these rubrics produces a side effect clients rarely expect: the organisation discovers where its own experts disagree. Two senior adjusters, same case, opposite calls. That finding predates the agent and outlives it. Resolving it improves the human operation immediately, and it is the kind of insight that only surfaces when someone finally writes the exam.
Ungradable is almost always a synonym for ungraded. The work of grading is real work. It is also the work that turns judgment from folklore into an asset the company owns.