RAG evaluation should separate retrieval failure from answer failure

RAG evaluation should separate retrieval failure from answer failure. I treat that separation as the minimum useful diagnostic boundary in a production system. An end-to-end score may tell me that the response was bad, but it does not tell me whether the index and retriever failed to surface the necessary evidence or whether the generator ignored, distorted, or overreached beyond evidence it received. Those are different defects, owned by different parts of the pipeline, and they demand different fixes.

I start with retrieval because the generator cannot use evidence it never sees. Recall@k tells me whether relevant material entered the candidate set, while Precision@k tells me how much irrelevant material came with it. MRR helps when the first useful result matters, and NDCG helps when graded relevance and ordering matter across the ranked list. I use these as retrieval diagnostics, not as proxies for answer quality. A retriever can rank the right passage highly and still feed a generator that produces an unfaithful answer.

The reverse is just as important. A weak final answer does not prove retrieval was weak. If the supplied context contains the answer, I evaluate whether the response is relevant and faithful to that context. RAGAS separates context precision and context recall from answer relevancy and faithfulness; ARES and TruLens provide related evaluation approaches. The tooling matters less than preserving the boundary. I want every failed case to answer two questions: was sufficient evidence available, and did the model use that evidence correctly?

This becomes essential once RAG stops being a single similarity search. Query routing can send a request to the wrong corpus. Query decomposition can omit a necessary sub-question. HyDE can produce a representation that retrieves plausible but misplaced material. Reranking can demote the best passage, contextual compression can remove the decisive sentence, and maximal marginal relevance can trade useful redundancy for diversity. Calling all of these “answer failures” erases the stage where the defect entered the system.

The same discipline clarifies several production pathologies. Embedding drift can make an index quietly less aligned with current queries. Context poisoning can place misleading evidence inside an otherwise relevant result set. Over-retrieval can bury the useful passage under superficially related text. These cases may all end in a fluent wrong answer, yet changing the generation prompt is not a serious response. I inspect the retrieved candidates, their order, and the context actually passed forward before touching the generator.

I also keep benchmark choice tied to the layer under test. BEIR and MS MARCO can exercise retrieval behavior, while context and answer measures can test the assembled RAG path. Adaptive or agentic RAG adds another evaluation surface: the system must decide whether to retrieve, where to route, whether to decompose, and when to stop. I log those decisions as pipeline outputs. Otherwise, a single aggregate score turns a sequence of inspectable choices into one opaque verdict.

There is one precise concession: an end-to-end score is useful as a release gate when the immediate question is whether the user-visible system improved. It is not sufficient for diagnosis. Once the gate fails, I need paired evidence—retrieval judgments for the candidate set and answer judgments against the supplied context. That pairing prevents teams from tuning generation around missing evidence or rebuilding an index to correct a model that already had what it needed.

My practical rule is to make evaluation mirror the data path. Test retrieval before generation, preserve the retrieved and compressed context, then score the answer against what was actually supplied. Slice failures by routing, decomposition, reranking, compression, drift, poisoning, and retrieval volume where those stages exist. RAG is a pipeline—its evaluation should retain enough structure to identify the broken component rather than merely announcing that the final response was wrong.