Retrieval Strategies: Beyond Naive RAG

The basic RAG pipeline — embed query, search vector DB, stuff results into prompt — works surprisingly well. Until it doesn't.

"What were the key changes in our Q3 pricing strategy compared to Q2?"

Naive RAG retrieves chunks about Q3 pricing AND chunks about Q2 pricing separately, but no single chunk contains the comparison. The LLM gets fragments and hallucinates the differences.

The query wasn't bad. The retrieval strategy was too simple.

This episode covers the advanced retrieval techniques that take RAG from "demo quality" to "production quality." Each one solves a specific failure mode of naive RAG.

Naive RAG: The Baseline

User Query → Embed → Vector Search (top-K) → Stuff into Prompt → LLM → Answer

When it works: Single-topic queries with direct matches in your corpus. When it fails:

Failure Mode Example Query Why It Fails

Let's fix each one.

Strategy 1: Multi-Query Retrieval

Problem: A single query embedding might not capture all aspects of what the user wants.

Solution: Use an LLM to generate multiple reformulations of the query, search with each, and merge results.

python

def multi_query_retrieve(original_query: str, k: int = 5):
# Step 1: Generate query variations
    prompt = f"""Generate 3 different versions of this search query
    to capture different aspects of the user's intent:
Original: {original_query}
Variations:"""
variations = llm.generate(prompt)
    # e.g., for "What changed in Q3 pricing vs Q2?"
    # → "Q3 pricing strategy changes"
    # → "Q2 vs Q3 pricing comparison"
    # → "pricing updates third quarter"
# Step 2: Search with each variation
    all_results = set()
    for query in [original_query] + variations:
        results = vector_search(query, k=k)
        all_results.update(results)
# Step 3: Deduplicate and rerank
    unique_results = list(all_results)
    reranked = rerank(original_query, unique_results, top_n=k)
return reranked

How it helps: The original query might miss Q2-specific chunks. The variation "Q2 vs Q3 pricing comparison" catches them. Multiple angles cast a wider net.

Cost: One extra LLM call for query generation (~50 tokens output). 3-4x more vector searches. Usually worth it.

🔧

Strategy 2: HyDE (Hypothetical Document Embeddings)

Problem: The query and the answer live in different "linguistic spaces." The user asks a question; the documents contain statements. Their embeddings might not be close.

Query: "What causes OOMKill in Kubernetes?" Document: "When a container exceeds its memory limit, the kernel terminates it with an Out-Of-Memory signal."

The question and the answer use different words. The embeddings might not be similar enough.

Solution: Generate a hypothetical answer to the query, then embed THAT instead of the query.

python

def hyde_retrieve(query: str, k: int = 5):
# Step 1: Generate a hypothetical answer
    prompt = f"""Answer this question with a detailed paragraph,
    even if you're not sure. Write it as if it's from a technical document:
{query}"""
hypothetical_doc = llm.generate(prompt)
    # "When a Kubernetes container exceeds the memory limit defined
    #  in resources.limits.memory, the Linux kernel's OOM killer
    #  terminates the process. This results in a CrashLoopBackOff..."
# Step 2: Embed the hypothetical document (not the query!)
    hyde_vector = embed(hypothetical_doc)
# Step 3: Search — hypothetical doc is in the same "space" as real docs
    results = vector_search(hyde_vector, k=k)
return results

Why it works: The hypothetical answer uses the same language patterns as your actual documents. "Container exceeds memory limit" → "container exceeds memory limit." The embedding of the fake answer is closer to the real answer than the embedding of the question.

Catch: If the LLM generates a wrong hypothetical answer, you search in the wrong direction. Works best when the LLM has some knowledge of the domain.

📊

Strategy 3: Parent-Child Chunks

Problem: Small chunks are great for precise retrieval but lack context. Large chunks have context but dilute the embedding.

Solution: Index small chunks (children) for search precision, but return large chunks (parents) for context.

python

During ingestion:

def create_parent_child_chunks(document: str):

# Parent chunks: 1000 tokens (context-rich)
    parent_splitter = RecursiveCharacterTextSplitter(chunk_size=1000)
    parents = parent_splitter.split_text(document)

# Child chunks: 200 tokens (search-precise)
    child_splitter = RecursiveCharacterTextSplitter(chunk_size=200)

all_children = []
    for parent_id, parent in enumerate(parents):
        children = child_splitter.split_text(parent)
        for child in children:
            all_children.append({
                "text": child,
                "parent_id": parent_id,
                "parent_text": parent,
            })

return parents, all_children

Index children in vector DB

Store parent mapping

During query:

def parent_child_retrieve(query: str, k: int = 5):
# Search against child chunks (precise)
    child_results = vector_search(query, collection="children", k=20)
# Get unique parent chunks
    parent_ids = set(r.payload["parent_id"] for r in child_results)
# Return parent chunks (contextual)
    parents = [get_parent(pid) for pid in list(parent_ids)[:k]]
    return parents
Parent (1000 tokens):

"Kubernetes resource management covers CPU and memory limits.

Resource limits define the maximum... Resource requests define

the minimum... LimitRange objects set defaults... ResourceQuota

limits total namespace consumption..."

Children (200 tokens each):

→ "Resource limits define the maximum CPU and memory a..."

→ "Resource requests define the minimum guaranteed..."

→ "LimitRange objects set defaults for a namespace..."

User searches: "what are resource limits?"

Match: Child #1 (precise match on "resource limits")

Return: Parent (full context about limits, requests, LimitRange)

Best of both worlds. Search precision of small chunks, context richness of large chunks.

Strategy 4: Contextual Retrieval (Anthropic's Method)

Problem: A chunk like "The company reported $4.2M revenue, up 23% YoY" means nothing without knowing which company, which quarter, which report.

Solution: Before embedding, prepend context from the full document to each chunk.

python

def add_context_to_chunks(document: str, chunks: list[str]):

contextualized_chunks = []

for chunk in chunks:
        prompt = f"""Here is the full document:
        {document[:5000]}

Here is a chunk from this document:
        {chunk}

Write a brief context (2-3 sentences) that situates this chunk
        within the full document. Include the document title, section,
        and any relevant context that would help understand this chunk
        in isolation."""

context = llm.generate(prompt)
        # "This chunk is from the Q3 2024 Earnings Report for Acme Corp.
        #  It appears in the Revenue section discussing quarterly performance."

contextualized_chunk = f"{context}\n\n{chunk}"
        contextualized_chunks.append(contextualized_chunk)

return contextualized_chunks

Before: "The company reported $4.2M revenue, up 23% YoY." After: "This is from Acme Corp's Q3 2024 Earnings Report, Revenue section. The company reported $4.2M revenue, up 23% YoY."

Impact: Anthropic reported 49% fewer retrieval failures with contextual retrieval. The context makes the chunk self-explanatory, so the embedding captures what it's actually about.

Cost: One LLM call per chunk during ingestion. For 10,000 chunks, that's ~$2-5 with a cheap model. But it's a one-time cost.

Strategy 5: Sentence Window Retrieval

Problem: You want the precision of sentence-level search but the context of surrounding text.

Solution: Index individual sentences, but return a window of surrounding sentences.

python

def sentence_window_retrieve(query: str, window_size: int = 3, k: int = 5):
# Search at sentence level (maximum precision)
    sentence_hits = vector_search(query, collection="sentences", k=k)
# Expand each hit to include surrounding sentences
    expanded_results = []
    for hit in sentence_hits:
        sentence_idx = hit.payload["sentence_index"]
        doc_id = hit.payload["doc_id"]
# Get window: N sentences before and after
        start = max(0, sentence_idx - window_size)
        end = sentence_idx + window_size + 1
window_sentences = get_sentences(doc_id, start, end)
        expanded_results.append(" ".join(window_sentences))
return expanded_results

The search finds the exact sentence. The window provides the surrounding context. Like using a highlighter with a magnifying glass.

🔬

Combining Strategies

The best RAG systems don't use just one strategy. Here's a production-ready combination:

User Query

↓

Multi-Query Generation (3 variations)

↓

For each query variation:

├── Hybrid Search (vector + BM25)
    └── Retrieve 20 candidates each
    ↓

Merge all candidates (deduplicate)

↓

Rerank with cross-encoder (top 5-10)

↓

Expand with parent chunks (if using parent-child)

↓

Build prompt with ranked, contextual chunks

↓

LLM generates answer

Not every application needs all of this. Start with naive RAG, measure what fails, add the strategy that fixes your specific failure mode.

🛡️

When to Use What

Strategy Solves Cost Complexity Use When

📦

Practical Takeaways

Start with naive RAG — it works for 60-70% of queries out of the box

Multi-query is the easiest upgrade — one extra LLM call, 20-30% better recall

Parent-child chunks give the best bang for buck — zero runtime cost, much better context

Contextual retrieval is worth the ingestion cost — 49% fewer failures (per Anthropic's data)

HyDE works magic for Q&A — but test it; wrong hypothetical answers hurt

Layer strategies incrementally — measure what's failing, add the fix, re-measure

🚀

What's Next?

Episode 26: Context Stuffing — You've retrieved the right chunks. Now how do you feed them to the LLM? Prompt templates, the "lost in the middle" problem, ordering strategies, and how to make the model actually USE the context you worked so hard to retrieve.

← Previous

Ep 24: Embedding Models

Next →

Ep 26: Context Stuffing

Next: Episode 26 — Context Stuffing

You retrieve 5 beautiful, relevant chunks. Then you stuff them into the prompt and the LLM ignores half of them. Here's why.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.