RAG Evaluation at Research Depth: Retrieval, Grounding, Faithfulness and Hallucination

RAG is not one model. It is a pipeline in which retrieval determines what evidence is available and generation determines how that evidence is transformed into an answer. A useful evaluation therefore decomposes the system rather than assigning one generic 'RAG score'.

The research literature consistently treats retrieval relevance, generation faithfulness and answer relevance as distinct dimensions. RAGAS introduced reference-free dimensions for RAG evaluation; ARES evaluates context relevance, answer faithfulness and answer relevance; RAGChecker further decomposes retrieval and generation into fine-grained diagnostics.

1. Model the RAG pipeline

A simplified pipeline is query → query transformation → retrieval → ranking → context construction → generation → post-processing.

Each stage creates a different failure mode. If the correct document is absent from the retrieved set, generation cannot reliably recover it. If the correct evidence is retrieved but ignored or contradicted, the generation stage is responsible. If the answer is supported but incomplete, the problem is coverage rather than basic faithfulness.

2. Evaluate retrieval independently

Build a retrieval test set with queries linked to relevant evidence. Measure recall@k, precision-oriented measures, reciprocal rank or other retrieval metrics appropriate to the search architecture.

Include paraphrases, multilingual variants, misspellings, long queries, ambiguous queries and adversarially similar documents.

A retrieval system that returns plausible but irrelevant passages can produce confident hallucinations because the generator has evidence-shaped noise.

3. Evaluate context utility

Retrieved relevance is not identical to context usefulness. Long contexts may contain the correct passage plus contradictory or distracting material.

Test whether the context contains sufficient evidence, whether evidence is duplicated, whether conflicting versions are present, and whether chunk boundaries preserve the facts required to answer the question.

4. Evaluate faithfulness at claim level

Break the answer into atomic factual claims and determine whether each claim is entailed by the retrieved evidence. This is more informative than judging the paragraph as a whole.

A response with nine supported claims and one unsupported claim should not be treated identically to a response in which every claim is unsupported.

RAGChecker's fine-grained design reflects this diagnostic philosophy and reports separate retrieval and generation behaviours rather than one opaque number.

5. Separate faithfulness from correctness

An answer can be faithful to an incorrect source. If the retrieved policy document is outdated, the generator may accurately reproduce obsolete information.

Therefore evaluate at least two dimensions: evidence support and external truth or source validity. Source freshness, authority, version and effective date should be part of the retrieval contract for regulated or policy-heavy domains.

6. Test answer completeness

Correctness is not sufficient when the task requires multiple facts. Define required fact sets or coverage criteria.

For example, a policy answer may need eligibility, deadline, required documents and exception rules. The response can be faithful yet fail because it omits the deadline.

7. Test abstention and unknown-answer behaviour

Construct questions whose answers are absent from the knowledge base. A safe RAG system should not manufacture support merely because the generator has a plausible prior.

Measure unsupported-answer rate on unanswerable questions and distinguish appropriate abstention from unnecessary refusal.

This is especially important because retrieval augmentation reduces but does not eliminate hallucination risk; the architecture still needs an explicit uncertainty policy.

8. Test contradiction and versioning

Insert controlled conflicts into the test corpus: old policy versus current policy, draft versus approved document, two regional variants, or two product versions.

The evaluator should verify that metadata and ranking rules select the authoritative source and that the answer does not merge incompatible versions.

9. Evaluate citations as evidence, not decoration

If the product exposes citations, verify that each citation actually supports the associated claim. Test citation completeness, citation correctness and source authority.

A citation that merely mentions the same topic is not evidence for the claim. The QA oracle should map claim → supporting passage → source metadata.

10. Use multiple evaluator types

Exact-match or symbolic checks are appropriate for structured facts. Retrieval metrics are appropriate for retrieval. Semantic graders are useful for claim entailment and answer quality. Human review remains important for calibrating subjective or domain-critical judgments.

LLM judges should be treated as measurement instruments requiring calibration, not as ground truth. Research on LLM judges demonstrates useful human alignment in some settings but also documents systematic biases.

11. Test robustness to irrelevant context

Add irrelevant passages, distractor documents and near-duplicate documents. The answer should remain grounded in the correct evidence.

This reveals whether the generator overweights recency, position, lexical overlap or misleading but fluent passages.

12. Test retrieval drift

Knowledge bases change. Re-indexing, chunking changes, embedding-model upgrades and ranking-model changes can alter the retrieved evidence without changing application code.

Maintain a fixed evaluation corpus and snapshot important retrieval results. Compare retrieval and answer metrics before and after index changes.

13. Research-grade RAG test matrix

The goal is diagnosis. A single hallucination percentage does not tell an engineering team whether to change retrieval, chunking, ranking, prompting, model selection or source governance. A decomposed evaluation does.