Production RAG: From Demo to Deployment

You built a RAG pipeline. It works on your laptop. You showed it to your team. Everyone clapped.

Now deploy it for 10,000 users hitting it simultaneously, with stale documents, inconsistent answers, zero observability, and a PM asking "why did the chatbot tell a customer we offer free shipping?" Good luck.

The gap between a RAG demo and production RAG is the same as the gap between cooking maggi at home and running a restaurant. The ingredients are similar. Everything else is different.

The Production RAG Stack

A demo RAG pipeline has 3 components: chunker → vector DB → LLM. A production RAG pipeline has about 15. Here's the full picture:

User Query

│
    ▼

┌──────────────┐

│ Query Router │ ── Cache hit? → Return cached answer

│ │ ── Route to correct index/collection

└──────────────┘

│
    ▼

┌──────────────┐

│ Query │ ── Rewrite, expand, decompose

│ Transform │ ── HyDE, step-back prompting

└──────────────┘

│
    ▼

┌──────────────┐

│ Retrieval │ ── Hybrid search (vector + keyword)

│ Layer │ ── Multi-index fan-out

└──────────────┘

│
    ▼

┌──────────────┐

│ Reranker │ ── Cross-encoder scoring

│ │ ── Filter by metadata/freshness

└──────────────┘

│
    ▼

┌──────────────┐

│ Context │ ── Assemble prompt

│ Assembly │ ── Citation tracking

└──────────────┘

│
    ▼

┌──────────────┐

│ LLM │ ── Generate answer

│ Generation │ ── Guardrails check

└──────────────┘

│
    ▼

┌──────────────┐

│ Post-process │ ── Fact verification

│ │ ── Format, log, return

└──────────────┘

Each box is a failure point. Each box needs monitoring. Welcome to production.

Caching: Don't Recompute What You Already Know

The single biggest win in production RAG is semantic caching. Most users ask the same questions in slightly different ways:

"What's the return policy?"

"How do I return something?"

"Can I get a refund?"

All the same intent. Without caching, each one triggers a full retrieval + LLM call. That's 3x the cost and latency for the same answer.

How Semantic Caching Works

python

import hashlib

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')
def get_cached_or_compute(query, cache, threshold=0.92):
query_embedding = model.encode(query)
# Search cache for semantically similar queries
    results = cache.search(query_embedding, top_k=1)
if results and results[0].score > threshold:
        # Cache hit — return stored answer
        return results[0].payload["answer"], "cache_hit"
# Cache miss — full RAG pipeline
    answer = run_full_rag_pipeline(query)
# Store in cache
    cache.insert(
        embedding=query_embedding,
        payload={"query": query, "answer": answer, "timestamp": now()}
    )
return answer, "cache_miss"

Cache Invalidation

The hardest problem in computer science, and it's 10x harder with semantic caches:

Strategy When to Use Tradeoff

Rule of thumb: Set TTL to half your document refresh frequency. If docs update daily, cache for 12 hours.

What Caching Gets You

Real numbers from production systems:

Cache hit rate: 40-70% after warm-up (higher for support bots, lower for research tools)

Latency reduction: 200ms cached vs 2-5s uncached (10-25x improvement)

Cost reduction: 50-70% on LLM API spend

Consistency: Same question → same answer (users notice inconsistency fast)

🔧

Document Versioning: Know What You're Serving

In production, your knowledge base isn't static. Documents get updated, deleted, replaced. You need to track what version of what document generated what answer.

The Versioning Problem

Monday: Policy v1 → "Returns within 30 days"

Tuesday: Policy v2 → "Returns within 14 days"

Wednesday: User asks about returns → Which version do they get?

If your re-indexing pipeline hasn't caught the v2 update, users get wrong answers. If it has, but your cache still holds the v1 answer, same problem.

Document Version Tracking

Every chunk in your vector store should carry metadata:

json

{

"chunk_id": "policy-returns-chunk-3",

"document_id": "policy-returns",

"document_version": "2024-03-15T10:30:00Z",

"content_hash": "sha256:a1b2c3...",

"source_url": "https://docs.company.com/returns",

"indexed_at": "2024-03-15T11:00:00Z",

"chunk_index": 3,

"total_chunks": 7

}

Re-indexing Strategies

Strategy How It Works Best For

Blue-green indexing is the gold standard for production. You never modify the live index — you build a new one and swap:

bash

Pseudo-workflow

1. Build new index: "products-v42"
2. Run eval suite against new index
3. If pass rate > threshold:
4. Alias "products-live" → "products-v42"
5. Keep old "products-v41" for 24h rollback
6. Else:
7. Alert team, keep "products-v41" live

📊

Monitoring: What Gets Measured Gets Fixed

A production RAG system without monitoring is like driving at night with no headlights. You'll crash. You just don't know when.

The Three Pillars of RAG Monitoring

1. Retrieval Quality

python

Track these metrics for every query

metrics = {
"retrieval_latency_ms": 145,
    "chunks_retrieved": 10,
    "chunks_after_rerank": 3,
    "top_chunk_relevance_score": 0.87,
    "source_documents_unique": 2,
    "retrieval_empty": False,  # CRITICAL — means no relevant docs found

}

1. Generation Quality

python

generation_metrics = {
"generation_latency_ms": 1200,
    "tokens_in": 2500,
    "tokens_out": 350,
    "answer_contains_citation": True,
    "answer_length_chars": 890,
    "guardrail_triggered": False,
    "model_used": "gpt-4o",

}

1. User Signals

python

user_metrics = {
"thumbs_up_down": "up",
    "follow_up_query": None,      # If user asks again → answer wasn't good
    "session_continued": True,     # Did they keep chatting or leave?
    "escalated_to_human": False,   # Worst case signal

}

Alerting Rules

Set up alerts for these production failure modes:

Alert Trigger Severity

A/B Testing RAG Pipelines

You want to try a new chunking strategy. Or a different embedding model. Or add a reranker. How do you know if it actually helps?

What to A/B Test

Component Variant A Variant B

Implementation Pattern

python

import random

def route_query(user_id, query):

# Deterministic assignment based on user_id
    # Same user always gets same variant (consistency)
    variant = "A" if hash(user_id) % 100 < 50 else "B"

if variant == "A":
        answer = pipeline_a(query)  # Current production
    else:
        answer = pipeline_b(query)  # Experimental

# Log everything for analysis
    log_experiment({
        "user_id": user_id,
        "variant": variant,
        "query": query,
        "answer": answer,
        "latency_ms": elapsed,
        "retrieval_scores": scores,
    })

return answer

Metrics That Matter

Don't just look at vibes. Measure:

Answer relevance — LLM-as-judge scoring (have GPT-4o rate answers 1-5)

Faithfulness — Does the answer stay grounded in retrieved context?

Latency — P50, P95, P99

Cost per query — Token usage × price

User satisfaction — Thumbs up/down ratio

Run for at least 1,000 queries per variant before drawing conclusions. Statistical significance matters.

Scaling: From 10 to 10,000 Queries Per Second

The Bottlenecks

In order of what usually breaks first:

Vector DB — Most vector DBs are single-node by default. At scale, you need sharding

LLM API — Rate limits, latency spikes, provider outages

Embedding computation — Real-time embedding of queries is fast, but batch re-indexing can saturate GPUs

Reranker — Cross-encoders are 10-100x slower than bi-encoders

Scaling Playbook

Bottleneck Solution

The Query Router Pattern

Not every query needs the full pipeline. Route intelligently:

python

def smart_route(query):

# Check cache first (fastest)
    cached = check_semantic_cache(query)
    if cached:
        return cached

# Classify query complexity
    complexity = classify_query(query)  # simple / moderate / complex

if complexity == "simple":
        # FAQ-style: small model, top-1 chunk
        return simple_pipeline(query, model="gpt-4o-mini", top_k=1)

elif complexity == "moderate":
        # Standard RAG
        return standard_pipeline(query, model="gpt-4o-mini", top_k=3)

else:
        # Complex: multi-step retrieval, powerful model
        return advanced_pipeline(query, model="gpt-4o", top_k=5, rerank=True)

This alone can cut costs by 60% — most queries are simple.

🔬

Common Failure Modes

Every production RAG system hits these. Know them before they find you.

1. The Stale Index Problem

Symptom: Users get outdated answers. Cause: Documents updated but index not re-built. Fix: Automated re-indexing pipeline + staleness alerts.

1. The Retrieval Miss

Symptom: AI says "I don't have information about that" when you know the doc exists. Cause: Chunking split the answer across two chunks. Neither chunk alone matches the query. Fix: Overlapping chunks, parent-child retrieval, or increase top-K.

1. The Context Poisoning

Symptom: AI gives confident wrong answers. Cause: Retrieved chunks contain contradictory or outdated information mixed with good chunks. Fix: Metadata filtering (recency), reranking, source deduplication.

1. The Prompt Injection via Documents

Symptom: AI behavior changes based on document content. Cause: A document contains text like "Ignore previous instructions and..." Fix: Sanitize documents at ingest, separate system prompt from retrieved context with strong delimiters.

1. The Infinite Loop

Symptom: Follow-up queries degrade in quality. Cause: Previous AI answers get fed back as context, compounding errors. Fix: Always retrieve from source documents, never from conversation history.

1. The Cost Spiral

Symptom: Your API bill doubles every month. Cause: Verbose system prompts + too many chunks + long conversations = token explosion. Fix: Prompt compression, aggressive caching, query routing to smaller models.

🛡️

Practical Takeaways

Semantic caching is your #1 cost and latency win — implement it before anything else

Version your documents and track what chunk generated what answer — you'll need this for debugging and compliance

Monitor retrieval quality, not just generation quality — garbage in, garbage out

A/B test pipeline changes with real traffic — lab evals miss production edge cases

Route queries by complexity — not every question needs your most expensive model

Plan for the six failure modes above — they're not hypothetical, they're inevitable

📦

What's Next?

Episode 29: What Is an Agent? — RAG retrieves and answers. Agents retrieve, plan, act, and use tools. We'll explore what makes an AI agent different from a chatbot, and why everyone is suddenly building them.

← Previous

Ep 27: RAG Evaluation

Next →

Ep 29: What Is an Agent

Next: Episode 29 — What Is an Agent

ChatGPT answers your questions. An agent does your work. That's the difference — and it changes everything.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.