RAG Evaluation Metrics: What to Measure and Why

Why RAG Systems Need Multi-Layer Measurement

A RAG system retrieves documents, then generates an answer conditioned on those documents. If retrieval misses the relevant passage, the generator cannot recover. If retrieval succeeds but the generator ignores the context or hallucinates, the answer fails anyway. If both components work but the system takes twelve seconds to respond, users abandon it.

Measuring only retrieval accuracy or only generation quality hides failure modes. A system with 95% retrieval recall but 40% faithfulness produces confident, wrong answers. A system with perfect faithfulness but 200ms p95 latency per retrieved chunk becomes unusable at scale. Effective evaluation tracks retrieval, generation, and system performance as separate layers, then measures end-to-end correctness.

Retrieval Metrics: Did the System Find the Right Documents?

Retrieval metrics measure whether the top-k results contain the information needed to answer the query. These metrics require a labeled test set where each query maps to known relevant documents.

Precision at K

Precision@k measures the fraction of retrieved documents that are relevant. If a system retrieves 5 documents and 3 are relevant, precision@5 is 0.60. High precision reduces noise passed to the generator, but says nothing about whether all relevant documents were found.

Recall at K

Recall@k measures the fraction of all relevant documents that appear in the top-k results. If 4 documents in the corpus are relevant and 3 appear in the top 5 results, recall@5 is 0.75. Recall matters more than precision in RAG because the generator can ignore irrelevant chunks, but cannot use chunks the retriever never surfaced.

Mean Reciprocal Rank (MRR)

MRR measures how high the first relevant document appears in the ranked list. If the first relevant document is at position 3, the reciprocal rank is 1/3. MRR averaged across queries gives a single score. Systems with high MRR surface useful context early, which matters when the generator weighs early chunks more heavily.

Normalized Discounted Cumulative Gain (NDCG)

NDCG accounts for both relevance and ranking position, with higher-ranked relevant documents contributing more to the score. NDCG requires graded relevance labels (e.g. not relevant, somewhat relevant, highly relevant), which makes it more expensive to label but more informative than binary metrics.

Generation Metrics: Did the Model Use the Context Correctly?

Generation metrics measure whether the language model produced an answer that is faithful to the retrieved documents and relevant to the query. These metrics typically require LLM-as-judge evaluation or human annotation.

Faithfulness (Groundedness)

Faithfulness measures whether every claim in the generated answer is supported by the retrieved documents. An answer is faithful if a human or evaluator LLM can trace each statement back to a specific passage. Unfaithful answers hallucinate facts not present in the context. Measuring faithfulness requires decomposing the answer into atomic claims and checking each one against the source documents.

Answer Relevance

Answer relevance measures whether the generated response actually addresses the user's query. A faithful answer can still be irrelevant if the model quotes a tangential passage instead of answering the question. Relevance is typically scored by an LLM judge that compares the query to the answer without seeing the retrieved context.

Context Relevance (Context Precision)

Context relevance measures whether the retrieved chunks were actually useful for generating the answer. If the generator only uses 2 of 10 retrieved chunks, context relevance is low, which indicates the retriever surfaced too much noise. This metric bridges retrieval and generation by showing whether the generator found the retrieval results useful.

End-to-End Metrics: Did the User Get the Right Answer?

End-to-end metrics measure the final output quality, independent of how the system produced it. These metrics align most closely with user experience.

Answer Correctness

Answer correctness measures whether the generated answer matches a known ground truth. For factual questions, this can be exact match or semantic similarity to a reference answer. For open-ended questions, LLM judges compare the generated answer to a reference on correctness and completeness. This is the metric that matters most to users, but it does not explain why a system failed.

Latency (p50, p95, p99)

Latency measures how long the system takes to return an answer. Track the 50th, 95th, and 99th percentile response times separately, because mean latency hides tail behavior. A system with 200ms p50 and 8-second p99 latency feels fast most of the time but fails unpredictably. Break latency into retrieval time, reranking time, and generation time to identify bottlenecks.

Cost per Query

Cost per query sums retrieval compute, embedding model inference, reranking, and LLM generation tokens. A system that retrieves 50 chunks and generates 500-token answers costs more than one that retrieves 5 chunks and generates 100 tokens. Cost matters at scale: multiply your provider's per-query rate by daily volume, and a gap that looks trivial on one query becomes the largest line in the budget.

Trade-Offs Between RAG Evaluation Metrics

These trade-offs explain why a single composite score (e.g. F1, weighted average) hides more than it reveals. A system with 0.85 average score might have 0.95 retrieval recall but 0.60 faithfulness, or 0.70 recall but 0.95 faithfulness. The first system needs a better generator; the second needs better retrieval. Tracking metrics separately surfaces which component to fix.

How to Measure RAG Evaluation Metrics in Practice

Measuring these metrics requires three assets: a test set of queries, ground truth labels, and an evaluation harness.

Build a Test Set

Start with 50-100 representative queries that span the distribution of questions your system will answer. Include edge cases: ambiguous queries, questions requiring multi-hop reasoning, queries where the answer is not in the corpus. Label each query with the IDs of relevant documents (for retrieval metrics) and a reference answer (for end-to-end metrics).

Choose Evaluation Methods

For retrieval metrics, compare retrieved document IDs to labeled relevant documents and calculate precision, recall, MRR, or NDCG. For generation metrics, use an LLM judge (GPT-4, Claude) to score faithfulness and relevance on a 1-5 scale, or decompose answers into claims and verify each one. For answer correctness, use semantic similarity (cosine similarity of embeddings) or LLM-based comparison to reference answers.

Automate and Version

Run evaluation automatically on every pipeline change. Track metrics over time in a dashboard or spreadsheet. Version your test set and evaluation code alongside your RAG pipeline so you can reproduce scores. When a metric regresses, the diff between pipeline versions shows what broke.

Measure Latency and Cost in Staging

Latency and cost metrics require running the system under realistic load. Measure p95 latency with 10-50 concurrent queries, not single-threaded execution. Calculate cost per query by logging token counts and retrieval operations, then multiplying by provider pricing.

When to Prioritize Each RAG Metric

Different use cases weight metrics differently. A customer support bot prioritizes answer correctness and latency over cost. A research assistant prioritizes recall and faithfulness over speed. An internal knowledge base prioritizes cost and faithfulness over answer diversity.

If users complain that answers are wrong, measure answer correctness and faithfulness first. If they complain that answers are incomplete, measure retrieval recall. If they complain that the system is slow, measure latency by component. If your infrastructure bill is growing faster than usage, measure cost per query and optimize retrieval chunk count or generation token limits.

Start by tracking answer correctness, retrieval recall@10, faithfulness, and p95 latency. Add other metrics when you need to diagnose a specific failure mode. A dashboard with twenty metrics is harder to act on than one with four.

Frequently Asked Questions

What is the most important RAG evaluation metric?

Answer correctness is the most important RAG evaluation metric because it measures whether the system delivered the right answer to the user. However, answer correctness alone does not explain why a system failed. Track retrieval recall, faithfulness, and latency alongside correctness so you can diagnose whether failures stem from missing documents, hallucination, or performance issues. A system with high answer correctness but low faithfulness is generating correct answers for the wrong reasons, which will break when the knowledge base changes.

How do you measure RAG faithfulness without human labeling?

Measure RAG faithfulness using an LLM judge that decomposes the generated answer into atomic claims, then checks whether each claim is supported by the retrieved context. Frameworks like RAGAS and TruLens automate this by prompting GPT-4 or Claude to score faithfulness on a scale or return a binary supported/unsupported judgment per claim. This approach correlates well with human judgment but adds an LLM call per evaluation, and that cost scales with answer length. For budget-constrained projects, sample 10-20% of answers for LLM-based faithfulness scoring rather than evaluating every query.

What is a good retrieval recall score for a RAG system?

A good retrieval recall@10 score for a RAG system is 0.80 or higher, meaning the top 10 retrieved chunks contain 80% of the relevant documents. Recall below 0.70 indicates the retriever is missing critical context, which no amount of prompt engineering can fix. Recall above 0.90 is achievable with dense retrieval models and reranking, but comes at higher latency and cost. The acceptable recall threshold depends on whether missing context causes silent failures (low tolerance) or graceful degradation (higher tolerance). Measure recall on a labeled test set that includes hard queries, not just easy ones.

Measure What Matters, Fix What Breaks

RAG evaluation metrics exist to surface failure modes, not to generate a score. A system with 0.90 recall, 0.85 faithfulness and 300ms p95 latency at a higher cost per query is not better or worse than one with 0.85 recall, 0.90 faithfulness, 500ms latency and a lower one. It is different, and the right choice depends on your use case.

Track retrieval, generation, and end-to-end metrics separately. Measure them on every pipeline change. When a metric regresses, the component-level breakdown tells you whether to fix retrieval, generation, or infrastructure. A RAG system without evaluation is a demo. A RAG system with the wrong evaluation metrics is a demo that thinks it is production.

Build RAG Systems That Pass Evaluation

WeaveAI builds production RAG systems with built-in evaluation harnesses that track retrieval recall, faithfulness, and answer correctness on every deployment. We measure what breaks before your users find it. See how we evaluate RAG pipelines at weaveai.dev/products/seo.