Every AI system looks like it works. That is the trap. A demo answers three questions correctly in a meeting and the room concludes the system is accurate, when what the room has actually seen is three data points chosen by the person giving the demo. If you are buying AI for your business, the most valuable skill you can acquire is knowing how accuracy is proven, because it changes which vendors survive your first meeting.
The method is not complicated. It has two parts with unglamorous names: ground truth and evals.
Ground truth: the answer key
Ground truth is a set of inputs paired with their verified correct outputs. For a system that reads invoices, it is a few hundred real invoices with every field checked by a person. For a system that answers questions from documents, it is a set of questions with known-correct answers and the passages that support them.
Two things about ground truth matter to a buyer.
It has to be built with your team. The people who do the work today are the only ones who know what a correct answer looks like in your business, including the awkward cases. A vendor who wants to build the test set without your people is testing against their own assumptions.
And it is hard work, which is where its value comes from. There is a paradox here: if perfect examples of the output already existed in quantity, you would not need the system. Assembling ground truth means collecting real cases, correcting them by hand and arguing about the edge cases. Skipping it defers the discovery of whether the system works until after the money is spent.
Evals: the scoring
An eval is an automated test that runs the system against the ground truth and scores the result. It runs continuously through the build rather than once at the end. Every change to the system gets scored: did accuracy go up or down, and on which kinds of case?
This is what turns AI development from guesswork into engineering. Without evals, week twelve of a project is opinion with confidence. With them, the team can see that a change improved date extraction but broke supplier matching, and act on it the same day.
The practice produces results that are hard to argue with. When the US firm Fractional AI built evals for Zapier’s documentation pipeline, its published write-up reported hallucinated API endpoints falling from 26% to under 1% once the team could measure what they were changing. The interesting part of that number is what it reveals: without the harness, nobody would have known the starting figure was 26%.
Evals also answer the model question. Businesses often worry about betting on the wrong AI vendor. A measured system dissolves the worry: when a better model ships, you run the evals against it and see, with numbers, whether the swap is safe. We build on Claude and stay model-agnostic for exactly this reason. The harness makes the claim checkable, which is how we run every build.
Confidence routing: what happens when it isn’t sure
Accuracy is never one hundred percent, and a well-built system is honest about that in its architecture. Each output carries a confidence score. High-confidence work completes automatically. Uncertain cases route to a person, with the source material and the doubt highlighted, and the person’s correction is stored so the system improves with use.
This matters to a buyer for two reasons. It means the failure mode is a person checking something, rather than a wrong answer flowing silently into your accounts. And it means the human review effort concentrates where the system is actually unsure, instead of someone spot-checking everything or nothing.
A related tell: ask what the system does when the answer is not in the source material at all. The right behaviour is to say so. A system that always produces an answer is a system that guesses, and for anything touching contracts, compliance documents or money, guessing is the dangerous property dressed up as a helpful one.
The questions to ask a vendor
Take these into any meeting where someone is selling you AI:
- What is the ground-truth set for this system, how large is it, and who built it?
- What accuracy does it score today, and on what kinds of case does it fail?
- Show me the eval results from your last change.
- What does it do when confidence is low? What does it do when the answer is absent?
- If we switch models next year, how would we know the swap is safe?
- Do we get the test set and the harness when the engagement ends?
A strong vendor enjoys these questions, because they are the questions their own process answers daily. A weak one will steer you back to the demo.
One caution in the other direction: a perfect score should raise your eyebrows, not settle them. “100% accurate” against an unstated test set is marketing. “Every answer in a set of 300 test questions traced to a correct source passage” is a measurement. The difference is whether the number comes with its denominator.
Why this is rare
Because demos close deals and harnesses do not, and because building ground truth is slow, human work in an industry selling speed. That scarcity is an opportunity for buyers. The vendors who measure are easy to identify, since they will show you the numbers before you ask, and the ones who cannot are easy to filter with one meeting’s worth of questions.
The demo is easy and production is the work. Ask for the score.