RAG Evaluation: How Do You Know If Your Pipeline Actually Works?

Your RAG pipeline is live. Users are asking questions. The LLM is answering. But is it answering correctly?

You can't manually check every response. You can't A/B test like a button color — wrong answers erode trust permanently. And the failure modes are subtle: the model confidently generates plausible-sounding answers that are completely wrong, grounded in the wrong chunk, or true but irrelevant to what was asked.

Traditional software testing doesn't apply here. assertEqual(response, expected) doesn't work when the output is natural language. You need new metrics, new frameworks, and a new way of thinking about "correct."

This is RAG evaluation. And if you're not doing it, you're flying blind.

The Three Dimensions of RAG Quality

Every RAG response can fail in three distinct ways. You need metrics for each:

User Question
                             ↓
                     ┌── RETRIEVAL ──┐
                     │ Did we find    │
                     │ the right      │  ← Context Relevance
                     │ documents?     │  ← Context Recall
                     └───────┬───────┘
                             ↓
                     ┌── GENERATION ──┐
                     │ Did the LLM    │
                     │ use them       │  ← Faithfulness
                     │ correctly?     │  ← Answer Relevance
                     └───────┬───────┘
                             ↓
                          Answer

1. Context Relevance (Retrieval Quality)

"Did we retrieve chunks that actually contain the answer?"

If your retrieval returns chunks about Kubernetes networking when the user asked about Kubernetes storage — game over. The LLM can't generate a correct answer from irrelevant context.

1. Faithfulness (Grounding)

"Is the answer actually supported by the retrieved context?"

The LLM might retrieve the right chunks but then ignore them and use its own training data. Or worse, it misinterprets the chunks and states something they don't support.

1. Answer Relevance

"Does the answer actually address the user's question?"

The answer might be faithful to the context and the context might be relevant, but the answer could still be off-topic, too vague, or answering a different question.

The RAGAS Framework

RAGAS (Retrieval Augmented Generation Assessment) is the leading open-source framework for RAG evaluation. It provides automated metrics for all three dimensions.

bash

pip install ragas

Setting Up Evaluation

python

from ragas import evaluate

from ragas.metrics import (

faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,

)

from datasets import Dataset

Your evaluation dataset

eval_data = {
"question": [
        "How do I set memory limits for Kubernetes pods?",
        "What is a CrashLoopBackOff?",
    ],
    "answer": [  # Your RAG pipeline's answers
        "To set memory limits, use resources.limits.memory in the pod spec...",
        "CrashLoopBackOff occurs when a container repeatedly fails to start...",
    ],
    "contexts": [  # Retrieved chunks (list of lists)
        ["Resource limits define the maximum CPU and memory a container can use..."],
        ["When a container crashes and Kubernetes restarts it repeatedly..."],
    ],
    "ground_truth": [  # Human-written correct answers
        "Set memory limits using resources.limits.memory field in the container spec...",
        "CrashLoopBackOff is a Kubernetes status indicating a container is crashing...",
    ],

}

dataset = Dataset.from_dict(eval_data)
results = evaluate(
dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],

)

print(results)

RAGAS Metrics Explained

How Each Metric Works

Faithfulness — RAGAS decomposes the answer into individual claims, then checks if each claim is supported by the context:

Answer: "Kubernetes uses 512Mi as the default memory limit.

Pods without limits can consume unlimited memory."

Claim 1: "Kubernetes uses 512Mi as the default memory limit"

→ Check context... NOT FOUND in context → Unfaithful!

Claim 2: "Pods without limits can consume unlimited memory"

→ Check context... SUPPORTED by context → Faithful

Faithfulness = 1/2 = 0.5

Context Precision — Measures if the relevant chunks are ranked higher than irrelevant ones:

Retrieved chunks:

1. [Irrelevant chunk about networking] ← Should be lower
2. [Relevant chunk about memory limits] ← Should be #1
3. [Relevant chunk about resource quotas] ← OK position

Context Precision penalizes having irrelevant chunks ranked above relevant ones.

🔧

Building an Evaluation Dataset

The hardest part of RAG evaluation: you need test data. Here's how to build it efficiently:

Method 1: Manual (Gold Standard)

python

eval_dataset = [
{
        "question": "How do I configure horizontal pod autoscaling?",
        "ground_truth": "Use kubectl autoscale or create an HPA resource with minReplicas, maxReplicas, and targetCPUUtilizationPercentage...",
        "source_doc": "kubernetes-guide.pdf, page 47",
    },
    # ... 50-100 examples covering key use cases

]

50-100 manually written question-answer pairs is the gold standard. Time-consuming but irreplaceable. Focus on:

Common user questions (check your logs)

Edge cases (ambiguous questions, multi-part queries)

Adversarial questions (questions your docs CAN'T answer)

Method 2: LLM-Generated (Scalable)

Use an LLM to generate question-answer pairs from your documents:

python

def generate_qa_pairs(chunk: str, n: int = 3) -> list[dict]:
prompt = f"""Based on this text, generate {n} question-answer pairs.
Questions should be natural, like a real user would ask.
    Answers should be directly derivable from the text.
    Include one question that the text does NOT fully answer.
Text:
    {chunk}
Generate {n} QA pairs in JSON format:
    [{{"question": "...", "answer": "...", "answerable": true/false}}]"""
return llm.generate(prompt, response_format="json")

Generate from your actual documents

all_qa = []

for chunk in sample_chunks(n=200):

qa_pairs = generate_qa_pairs(chunk)
    all_qa.extend(qa_pairs)

IMPORTANT: Human review a sample (20-30%) to validate quality

Method 3: Production Logging (Continuous)

Log real user queries and periodically evaluate responses:

python

def log_rag_interaction(query, retrieved_chunks, answer):

log_entry = {
        "timestamp": datetime.now().isoformat(),
        "query": query,
        "chunks": retrieved_chunks,
        "answer": answer,
        "evaluated": False,
    }
    append_to_log(log_entry)

Weekly: sample 50 logged interactions, have a human rate them

This builds your evaluation dataset over time

📊

Automated Evaluation Pipeline

Run evaluation as part of your CI/CD:

python

def evaluate_rag_pipeline(eval_dataset: list[dict]) -> dict:
"""Run the full RAG pipeline on eval data and compute metrics."""
results = []
    for example in eval_dataset:
        # Run your actual RAG pipeline
        rag_output = answer_question(example["question"])
results.append({
            "question": example["question"],
            "answer": rag_output["answer"],
            "contexts": rag_output["chunks"],
            "ground_truth": example["ground_truth"],
        })
# Compute RAGAS metrics
    dataset = Dataset.from_list(results)
    scores = evaluate(dataset, metrics=[
        faithfulness, answer_relevancy, context_precision, context_recall
    ])
return {
        "faithfulness": scores["faithfulness"],
        "answer_relevancy": scores["answer_relevancy"],
        "context_precision": scores["context_precision"],
        "context_recall": scores["context_recall"],
        "n_evaluated": len(results),
        "timestamp": datetime.now().isoformat(),
    }

Setting Quality Thresholds

python

QUALITY_THRESHOLDS = {

"faithfulness": 0.85,         # Answers must be grounded
    "answer_relevancy": 0.80,     # Answers must be on-topic
    "context_precision": 0.75,    # Retrieved chunks must be relevant
    "context_recall": 0.80,       # Must retrieve the right information

}

def check_quality_gate(scores: dict) -> bool:
"""Return True if all metrics pass thresholds."""
    for metric, threshold in QUALITY_THRESHOLDS.items():
        if scores[metric] < threshold:
            print(f"FAIL: {metric} = {scores[metric]:.3f} (threshold: {threshold})")
            return False
    return True

Human Evaluation: When Automated Isn't Enough

Automated metrics are useful but imperfect. They use an LLM to judge another LLM — there's inherent circularity. Human evaluation is essential for:

Tone and style — Is the answer appropriate for your audience?

Completeness — Did it answer ALL parts of the question?

Harmful outputs — Does it give dangerous advice?

Edge cases — How does it handle ambiguous or adversarial queries?

A Simple Human Evaluation Rubric

For each RAG response, rate 1-5:

1. CORRECTNESS: Is the factual content accurate?

1 = Wrong  2 = Mostly wrong  3 = Partially correct  4 = Mostly correct  5 = Perfect

1. COMPLETENESS: Does it fully answer the question?

1 = Missing  2 = Bare minimum  3 = Adequate  4 = Thorough  5 = Complete

1. GROUNDEDNESS: Is it based on the retrieved documents?

1 = Hallucinated  2 = Mostly made up  3 = Mixed  4 = Mostly grounded  5 = Fully grounded

1. HELPFULNESS: Would a user find this useful?

1 = Useless  2 = Barely helpful  3 = Acceptable  4 = Helpful  5 = Excellent

Overall Score = Average of all four

How many to evaluate: Sample 50-100 responses per evaluation cycle. Prioritize:

Low-confidence responses (if your pipeline outputs confidence scores)

Long/complex responses (more room for error)

Random sample (avoid selection bias)

What "Good" Looks Like

Based on industry benchmarks and practical experience:

Metric Poor Acceptable Good Excellent

If faithfulness is below 0.8, fix it before anything else. A system that confidently gives wrong answers is worse than no system at all.

🔬

Practical Takeaways

Evaluate three dimensions — retrieval quality (context), grounding (faithfulness), and answer quality (relevance)

RAGAS is the fastest way to start — automated metrics that work out of the box

Build a 50-100 example eval set — manually or LLM-generated, human-reviewed

Run eval in CI/CD — catch regressions before they reach users

Human eval is irreplaceable — do it monthly on 50-100 samples

Faithfulness > everything — a wrong answer is worse than "I don't know"

🛡️

What's Next?

Episode 28: Production RAG — Evaluation tells you the pipeline works. Now ship it. Caching, versioning, monitoring, A/B testing, scaling, and the common failure modes that only appear in production.

← Previous

Ep 26: Context Stuffing

Next →

Ep 28: Production RAG

Next: Episode 28 — Production RAG

You built a RAG pipeline. It works on your laptop. Everyone clapped. Now put it in production. That's where the real work begins.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.