Skip to content
Research note · TechnologySerial number MSK–R/002

Evaluating RAG systems is harder than it looks.

Why retrieval quality, not the model, decides whether your system tells the truth.

Written by
Ana Pérez · Maistik Studio
Published
Reading time
3 min read

Technical sheet

MethodRetrieval and generation scored separately, offline then online
DatasetReal support tickets and search logs, labelled with the client team
MetricPassage recall, answer correctness and correct refusals
CadenceWeekly, on a fixed question set

Abstract

Retrieval quality, not the model, decides whether a RAG system tells the truth. We score retrieval and generation apart, on real questions, on a schedule, and treat a correct refusal as a result.

Key findings

  1. Most wrong answers start in retrieval; the model writes fluently about whatever it is given.
  2. A refusal on a missing source is a correct result and has to be scored as one.
  3. A score from launch week says nothing about autumn; the schedule is the evaluation.
On this page
  1. Methodology.
  2. Results.
  3. Limits.
  4. Conclusions.

Methodology.

Everyone tests the model. Almost nobody tests the retrieval step with the same rigor, and retrieval is where most of the wrong answers come from. So we separate two questions that get lumped into one: did we retrieve the right passages, and did we answer correctly given what we retrieved.

We build the evaluation set from real questions, not invented ones: support tickets, search logs, the questions your team already gets asked. We label the passages that should come back for each question, so retrieval has its own score, separate from generation.

Results.

A RAG system can pass every spot check a person runs by hand, because a person tends to ask questions the system is good at, on documents the system already indexed well. The gaps show up on the questions nobody thought to ask.

Once retrieval and generation are measured separately, the fixes get cheaper and faster to find. A chunking problem stops looking like a model problem. A stale index stops looking like a hallucination you cannot explain.

We also test what happens when the answer is not in the documents. A system that guesses instead of saying “I don’t have that” is worse than no system, because it hides its own failure behind good grammar. Refusing correctly is a result we measure, not an afterthought.

Limits.

None of this removes the need for judgment. Someone still has to decide what “correct” means for your documents, your customers and your risk tolerance. We build the evaluation with your team, not for it, because that judgment does not transfer well from one company to another.

Conclusions.

We run the evaluation on a schedule, not once at launch. New documents arrive, old ones get retired, and a phrasing that used to retrieve the right passage can quietly stop working. Treating evaluation as a one-time gate is how a system that passed every test in March starts giving wrong answers by autumn, with nobody able to say when it began.

References
  1. Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, 2020.
  2. Es et al., “RAGAS: Automated Evaluation of Retrieval Augmented Generation”, 2023.
Related

Keep reading.

Network of connections spreading from Valencia across the globe

Working on something *like this*?

Tell us what you are building. We reply with a plain answer on whether AI is the right tool.

Talk to usor write to hello@maistik.studio