How to Evaluate RAG: Score Retrieval and Generation Separately

To evaluate RAG, score its two halves separately. Retrieval gets graded on whether the right documents come back, using recall@k and MRR against a labeled test set. Generation gets graded on whether the answer is faithful to those documents and correct. Build 50 to 100 test questions with known sources, run both scores on every change, and the numbers tell you which half is broken.

Most teams do the opposite. They ask a handful of questions, read the answers, and decide it "seems good." Then they change the chunk size, ask a different handful, and decide it still seems good. Nobody can say whether the change helped, because there was never a number to move.

01TL;DR

• Two scorecards, not one. A wrong answer is either a retrieval miss or a generation miss, and the fixes have nothing in common.
• Build a golden set first. 50 to 100 real questions, each tagged with the document that answers it, plus about 10 percent that have no answer.
• Retrieval: recall@k and MRR. Cheap, deterministic, and no LLM needed to compute them.
• Generation: faithfulness and correctness. Use an LLM judge that checks claims one at a time, calibrated against your own labels.
• Run it on every change. A chunk size, a prompt edit, or a new embedding model should never ship without a before and after.

02Why RAG needs two scorecards

A RAG pipeline is a search engine bolted to a writer. The retriever finds chunks, the model writes an answer from them. When a user gets a bad answer, the cause sits in one of those two places.

If the retriever never returned the refund policy, no prompt tweak will make the model quote it correctly. If the retriever did return it and the model still invented a 60-day window, re-chunking is wasted work. One blended "answer quality" score hides that difference, so you end up fixing the wrong half.

03Step 1: Build the golden set

The golden set is a list of questions with known right answers and known source documents. It is the whole foundation, and it is the step people skip.

Pull questions from three places, in this order of value:

1. Real user queries. Search logs, support tickets, Slack questions. These carry the messy phrasing your system actually sees.
2. Questions from the person who owns the docs. Ask them for 20 questions they get asked every month, and 5 they consider tricky.
3. Synthetic questions. Have a model write one question per sampled chunk, then delete the ones a real person would never ask. Synthetic questions skew toward copying the chunk's wording, which makes retrieval look better than it is, so keep them under half the set.

Then add about 10 percent questions your documents cannot answer. "What is your phone support number?" when you only offer email. These measure whether the system says "I don't know" or makes something up.

Each entry looks like this:

{
  "id": "q-031",
  "question": "Can I get a refund after 30 days on an annual plan?",
  "expected_docs": ["billing/refund-policy"],
  "reference_answer": "No. Annual plans are refundable within 30 days of purchase, prorated after that only if the account was charged in error.",
  "answerable": true
}

Label sources by document ID, not chunk ID. Chunk IDs change every time you re-chunk, and you will re-chunk. A document-level label survives that, so your old test set still works on your new chunking strategy.

04Step 2: Score retrieval

Run each question through the retriever only. No model call. Compare the returned document IDs to expected_docs.

• Recall@k: the share of questions where at least one expected document is in the top k results. If k is 5 and recall@5 is 0.82, the right source made it into the context 82 percent of the time.
• MRR (mean reciprocal rank): for each question, 1 divided by the rank of the first correct result, averaged. First place scores 1.0, third place scores 0.33, missing scores 0. It tells you whether right answers come back near the top or barely scrape in.

def score_retrieval(golden, retrieve, k=5):
    hits, rr = 0, 0.0 for q in golden:
        if not q["answerable"]:
            continue docs = [c.doc_id for c in retrieve(q["question"], k=k)] ranks = [i + 1 for i, d in enumerate(docs) if d in q["expected_docs"]] if ranks:
            hits += 1 rr += 1 / ranks[0] n = sum(q["answerable"] for q in golden) return {"recall_at_k": hits / n, "mrr": rr / n}

This runs in seconds and costs nothing beyond embedding the queries. It is the number to watch when you change chunk size, swap embedding models, add hybrid keyword search, or bolt on a reranker.

As a working rule, I want recall@5 above 0.85 before I spend any time on the generation prompt. Below that, the model is being asked to answer from the wrong pages, and nothing downstream can fix it.

05Step 3: Score generation

Now run the full pipeline and grade the answers. Four checks matter:

• Faithfulness. Every claim in the answer is supported by the retrieved chunks. This is your hallucination meter.
• Correctness. The answer agrees with the reference answer.
• Citation accuracy. The cited source actually contains the cited claim.
• Refusal. On unanswerable questions, the system declines instead of guessing.

Faithfulness and correctness need judgment, so use an LLM as a judge. The trick that makes it reliable is checking claims one at a time instead of asking for an overall score:

You are grading an answer against source passages.
1. Split the ANSWER into individual factual claims.
2. For each claim, output SUPPORTED if a SOURCE states it, or UNSUPPORTED if no SOURCE states it.
3. Do not use outside knowledge. A true claim that is not in the SOURCES is UNSUPPORTED.
Return JSON: {"claims": [{"claim": "...", "verdict": "..."}]}

Faithfulness is then the share of SUPPORTED claims. A 1 to 10 "how faithful is this" score drifts from run to run. Per-claim verdicts are far more stable and they show you the exact sentence that went wrong.

Before trusting the judge, hand-label about 30 answers yourself and compare. If it agrees with you less than roughly 85 percent of the time, tighten the rubric or move to a stronger judge model. Unless you check, you're trusting a vibe with a number on it.

06Read the failure matrix

With both scores per question, every failure lands in one cell of a simple grid. This is where the evaluation pays off, because each cell has a different fix.

The bottom-left cell catches people out. The retriever missed, but the answer was right anyway, because the model knew it from training. That looks like a pass, and it is a liability: the same model will confidently "know" things that are out of date or wrong for your company. Treat it as a grounding failure.

On a real project, the counts usually make the decision obvious. If 14 of 20 failures are retrieval misses, the next week goes to chunking and reranking, and the prompt stays untouched.

07Tools vs a script

Ragas, DeepEval, TruLens, and promptfoo all ship ready-made RAG metrics such as faithfulness, context recall, and answer relevance. They save setup time and give you a dashboard.

I'd still start with a script. The retrieval function above, one judge prompt, and a CSV of results is about 100 lines, and you will understand every number it produces. Move to a framework once you know which three metrics you actually look at each week. If you are building agents on top of retrieval, my AI Agents 101 resource covers where evals fit in the wider build.

08Run it on every change, then in production

An eval set you run once is a demo. Wire it into the work:

• Before and after every change. Chunk size, embedding model, top-k, prompt, model version. Log both scores next to the commit.
• Fail the build on regressions. A drop of more than 3 points in recall@5 or faithfulness blocks the merge. Same idea as testing an AI agent before shipping.
• Sample production. Run the faithfulness judge on 2 to 5 percent of live answers each day. You won't have reference answers there, but faithfulness only needs the retrieved context.
• Feed failures back. Every thumbs-down or escalated ticket becomes a candidate golden question. A set that grows from real misses gets sharper every month.

The cost is small. 100 questions with a judge reading about 3,000 tokens each is around 300,000 input tokens per run, which is a dollar or two at typical mid-tier pricing. Retrieval scoring is close to free.

09Mistakes that make RAG evals lie

Testing only on questions you wrote while reading the docs. You copy their vocabulary, so retrieval looks great. Real users say "cancel" when the doc says "terminate."

Skipping unanswerable questions. Without them you never measure the most damaging behavior, which is a confident answer with no source.

Changing the test set and the system at once. Freeze the set for a comparison. Add new questions afterward, then re-baseline.

Averaging away the tail. A 0.90 overall score can hide a whole category, such as pricing questions, sitting at 0.40. Tag questions by topic and read the scores per tag.

10The bottom line

RAG evaluation comes down to one move: stop grading the pipeline as a single thing. Score retrieval with recall@k and MRR, score answers with a per-claim faithfulness judge you have checked against your own labels, and sort every failure into the matrix before touching any code.

Tonight, pull 50 real questions from your logs, tag each with the document that answers it, and run the retrieval function above. That single recall@5 number will tell you more about your system than a week of reading answers by eye.

I send builds and breakdowns like this in the newsletter, if you want the next one.