We build instead of wrap. Here is why.
A wrapper gets you a demo in a week. A system gets you an answer you can defend.
PlaybookSerial number MSK–P/005
How to evaluate an AI system before you trust it.
Trust is a number you measured, not a feeling you had in the demo.
By Ana Pérez3 min read
The demo is the worst possible evaluation.
A demo is run by the person who built the system, on inputs they chose, in front of people who want it to work. Every one of those conditions is missing on a normal Tuesday. The system will meet a document scanned sideways, a question phrased by someone in a hurry and a case nobody thought to include. If the only evidence you have is the demo, you have no evidence.
That does not mean the system is bad. It means you do not know yet, and the point of an evaluation is to replace “we do not know” with a number and a list of the cases behind it.
Build the evaluation set from your own work
Take the questions your team already gets asked. Take the documents they actually search. Take the decisions they already make by hand, and write down what the right outcome was. Fifty real cases are worth more than five thousand invented ones, because the invented ones will be shaped by the same assumptions the system was built on.
Label what correct means before you run anything
Do it case by case, before a single run. For a document reader, correct is the passage the answer should come from. For a detector, it is the reading a technician would have flagged. For an agent, it is the action a colleague would have taken, and the actions they would never take.
Measure the parts separately
A system that answers wrongly can fail in the retrieval, in the model or in the rules around it. If you only score the final answer, you cannot tell which. Score each part on its own: did the right passages come back, did the model reason correctly over what it was given, did the guardrails hold when the answer was not there.
Score refusals as results
A system that says “I do not have that” when the source is missing is behaving correctly, and it should be scored as correct. A system that guesses fluently is failing, however good the grammar. Count both, and read the two numbers together.
Keep measuring after launch
The inputs change under a system. New suppliers, new phrasing, a template somebody edited. A score that was fine at launch tells you nothing about autumn. Run the same evaluation on a schedule, watch the number, and treat a drop the way you would treat an outage.
Working on something *like this*?
Tell us what you are building. We reply with a plain answer on whether AI is the right tool.
Talk to usor write to hello@maistik.studio