Evaluation and assurance

An AI system without an evaluation suite has no definition of done. Accuracy described in a demo is an anecdote. A suite written against real cases, in the language of the business, is something closer to a contract, and it is the only version of the claim that survives contact with production.

Assurance is also what makes change safe. Models get replaced underneath you, prompts are edited, retrieval shifts as the corpus grows. Evaluation is what tells you the system moved before your users are the ones telling you.

What we usually find instead is confidence resting on the wrong evidence. A model is chosen on a public benchmark that has nothing to do with your cases. Quality is reported as a single number nobody can trace to a decision anyone cares about. The people who actually know what a right answer looks like, the underwriters, the clinicians, the controllers, are consulted at the demonstration and never again. Then the system is wrong in front of a customer, and there is no way to establish whether it was always wrong, whether something changed underneath it, or whether the case was simply unusual. The argument becomes a matter of opinion between the business and the team that built the thing.

So we write the evaluation suite before we build, with the people who will be held responsible for the output. The cases come from your own history, including the ones that went badly, and each expected answer is recorded in the words the business would use to defend it. Failures are separated by kind, because a retrieval miss, a reasoning error and a formatting error are three different repairs. The suite then runs on every change, so a new model version, an edited prompt or a reindexed corpus becomes something that has to pass rather than something someone hopes is harmless.

Assurance goes further than accuracy. We record what the system was asked, what it retrieved, what it did and under whose identity, because an answer nobody can reconstruct is an answer a regulator or an auditor will not accept. We also write down what the system is not permitted to decide alone, and put a person on that path by design rather than as an apology after an incident. What the business ends up holding is a claim it can make out loud: not that the system is intelligent, but that it is right on the cases that matter, that the evidence is on file, and that anyone can see the moment it stops being right.