Ep 22: RAG Pipeline End-to-End: From Document to Answer

You've learned the pieces: embeddings, vector databases, chunking, hybrid search, reranking. Now let's build the whole machine.

RAG — Retrieval-Augmented Generation — is the pattern that makes LLMs useful for your specific data. Instead of hoping the LLM memorized the answer during training, you retrieve relevant documents and stuff them into the prompt. The LLM generates an answer grounded in your actual data.

No fine-tuning. No retraining. Just smart retrieval + prompt engineering.

This is the episode where everything clicks together.

The Two Phases

A RAG system has two distinct phases that run at very different times:

PHASE 1: INGESTION (offline, batch)

─────────────────────────────────────

Documents → Parse → Chunk → Embed → Store in Vector DB

(runs once per document, or on update)

PHASE 2: QUERY (online, real-time)

─────────────────────────────────────

User Question → Embed → Search → Rerank → Build Prompt → LLM → Answer

(runs every time a user asks something)

⚡

Phase 1: Ingestion Pipeline

Step 1: Document Collection

Gather your source documents. This could be:

Sources:

├── Confluence pages (HTML)

├── Google Docs (API export)

├── PDFs (contracts, manuals)

├── GitHub repos (code + docs)

├── Slack/Discord messages (JSON)

├── Database records (SQL export)

└── Web pages (scraping)

Step 2: Document Parsing

Convert raw formats into clean text. This is harder than it sounds (more in Episode 23).

python

from langchain.document_loaders import (

example
code
PyPDFLoader,
    UnstructuredHTMLLoader,
    TextLoader,
    CSVLoader,

)

Load different formats

snippet
code
pdf_docs = PyPDFLoader("manual.pdf").load()
html_docs = UnstructuredHTMLLoader("page.html").load()
text_docs = TextLoader("readme.md").load()

Step 3: Chunking

Split documents into retrieval-friendly chunks (see Episode 19):

python

from langchain.text_splitter import RecursiveCharacterTextSplitter

snippet
code
splitter = RecursiveCharacterTextSplitter(
chunk_size=500,       # ~500 characters per chunk
    chunk_overlap=50,     # 10% overlap
    length_function=len,

)

snippet
code
chunks = []

for doc in all_documents:

example
code
doc_chunks = splitter.split_documents([doc])
    chunks.extend(doc_chunks)

snippet
code
print(f"Total chunks: {len(chunks)}")  # e.g., 15,432

Step 4: Embedding

Convert each chunk into a vector:

python

from openai import OpenAI

snippet
code
client = OpenAI()
def embed_batch(texts, model="text-embedding-3-small"):
response = client.embeddings.create(model=model, input=texts)
    return [r.embedding for r in response.data]

Embed in batches (API limits)

snippet
code
batch_size = 100
all_embeddings = []

for i in range(0, len(chunks), batch_size):

example
code
batch = [chunk.page_content for chunk in chunks[i:i+batch_size]]
    embeddings = embed_batch(batch)
    all_embeddings.extend(embeddings)

Or with Ollama (free, local):

bash

Using nomic-embed-text locally

curl http://localhost:11434/api/embeddings -d '{

"model": "nomic-embed-text",

"prompt": "Your chunk text here"

Step 5: Store in Vector Database

python

from qdrant_client import QdrantClient, models

snippet
code
client = QdrantClient("localhost", port=6333)

Create collection

client.create_collection(

example
code
collection_name="knowledge_base",
    vectors_config=models.VectorParams(
        size=1536,                    # Match your embedding model
        distance=models.Distance.COSINE,
    ),

)

Upload points

snippet
code
points = []

for i, (chunk, embedding) in enumerate(zip(chunks, all_embeddings)):

example
code
points.append(models.PointStruct(
        id=i,
        vector=embedding,
        payload={
            "content": chunk.page_content,
            "source": chunk.metadata.get("source", "unknown"),
            "page": chunk.metadata.get("page", None),
            "chunk_index": i,
        }
    ))

Batch upload

client.upsert(collection_name="knowledge_base", points=points)

snippet
code
print(f"Uploaded {len(points)} chunks to Qdrant")

The Complete Ingestion Pipeline

python

def ingest_documents(file_paths: list[str]):

example
code
"""Full ingestion pipeline."""
    # 1. Load
    documents = []
    for path in file_paths:
        loader = get_loader(path)  # Pick loader by file type
        documents.extend(loader.load())

example
code
# 2. Chunk
    splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
    chunks = splitter.split_documents(documents)

example
code
# 3. Embed
    texts = [c.page_content for c in chunks]
    embeddings = embed_batch(texts)

example
code
# 4. Store
    points = [
        models.PointStruct(id=i, vector=emb, payload={
            "content": chunk.page_content,
            "source": chunk.metadata["source"],
        })
        for i, (chunk, emb) in enumerate(zip(chunks, embeddings))
    ]
    qdrant.upsert(collection_name="kb", points=points)

example
code
return len(points)

🔧

Phase 2: Query Pipeline

Step 1: Embed the Query

python

snippet
code
query = "How do I set memory limits for Kubernetes pods?"
query_vector = embed(query)  # Same model as ingestion!

Critical: Use the same embedding model for queries as you used for documents. Mixing models (e.g., embedding docs with nomic-embed but querying with text-embedding-3) produces vectors in different spaces — cosine similarity becomes meaningless.

Step 2: Retrieve Candidates

python

Vector search

snippet
code
candidates = qdrant.search(
collection_name="knowledge_base",
    query_vector=query_vector,
    limit=20,  # Get more than you need for reranking

)

For hybrid search (vector + BM25):

python

snippet
code
candidates = qdrant.query_points(
collection_name="knowledge_base",
    prefetch=[
        models.Prefetch(query=query_vector, using="dense", limit=20),
        models.Prefetch(query=query_text, using="sparse", limit=20),
    ],
    query=models.FusionQuery(fusion=models.Fusion.RRF),
    limit=20,

)

Step 3: Rerank

python

import cohere

snippet
code
co = cohere.Client("YOUR_KEY")
docs = [hit.payload["content"] for hit in candidates]
reranked = co.rerank(
model="rerank-english-v3.0",
    query=query,
    documents=docs,
    top_n=5,

)

Get top 5 reranked chunks

snippet
code
top_chunks = [candidates[r.index].payload["content"] for r in reranked.results]

Step 4: Build the Prompt

python

snippet
code
context = "\n\n---\n\n".join(top_chunks)
prompt = f"""Answer the user's question based on the following context.

If the context doesn't contain enough information, say so.

Context:

{context}

Question: {query}

Answer:"""

Step 5: Generate the Answer

python

from openai import OpenAI

snippet
code
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a helpful assistant. Answer based on the provided context only."},
        {"role": "user", "content": prompt}
    ],
    temperature=0.1,  # Low temp for factual answers

)

snippet
code
answer = response.choices[0].message.content

The Complete Query Pipeline

python

snippet
code
def answer_question(query: str) -> dict:
"""Full RAG query pipeline."""
    # 1. Embed query
    query_vector = embed(query)
# 2. Retrieve candidates
    candidates = qdrant.search(
        collection_name="kb",
        query_vector=query_vector,
        limit=20,
    )
# 3. Rerank
    docs = [hit.payload["content"] for hit in candidates]
    reranked = co.rerank(
        model="rerank-english-v3.0",
        query=query, documents=docs, top_n=5,
    )
    top_chunks = [docs[r.index] for r in reranked.results]
# 4. Build prompt
    context = "\n\n---\n\n".join(top_chunks)
    prompt = f"Context:\n{context}\n\nQuestion: {query}\n\nAnswer:"
# 5. Generate
    response = llm.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Answer based on context only."},
            {"role": "user", "content": prompt},
        ],
    )
return {
        "answer": response.choices[0].message.content,
        "sources": [candidates[r.index].payload["source"] for r in reranked.results],
        "chunks_used": len(top_chunks),
    }

📊

The Full Architecture

example
code
INGESTION (offline)

╔══════════════════════════════════════════════════════╗

║ Documents → Parser → Chunker → Embedder → VectorDB ║

╚══════════════════════════════════════════════════════╝

example
code
QUERY (real-time)

╔══════════════════════════════════════════════════════╗

║ Question → Embedder → VectorDB → Reranker → ║

║ → Prompt Builder → LLM → Answer ║

╚══════════════════════════════════════════════════════╝

Latency Breakdown

Step Time Notes

Query embedding	20-50ms	API call to embedding model
Vector search	5-20ms	HNSW approximate search
Reranking	100-300ms	Cross-encoder on 20 candidates
Prompt building	<1ms	String concatenation
LLM generation	500-3000ms	Depends on model and output length
Total	~700-3500ms	Most time spent on LLM generation

Cost Breakdown (per query, approximate)

Step Cost Provider

Query embedding	$0.00002	OpenAI text-embedding-3-small
Vector search	$0.0001	Self-hosted Qdrant
Reranking (20 docs)	$0.00004	Cohere Rerank
LLM (1K input + 500 output)	$0.005	GPT-4o
Total per query	~$0.005	Dominated by LLM cost

Common Mistakes

Mistake Impact Fix

Different embedding models for ingestion and query	Zero relevant results	Always use the same model
No chunking overlap	Lost context at boundaries	10-15% overlap
Retrieving too few candidates	Missed relevant docs	Retrieve 20-50, rerank to top 5
Stuffing too many chunks in prompt	Confuses LLM, "lost in the middle"	3-5 chunks is usually optimal
No source attribution	Users can't verify answers	Always return source metadata
Skipping reranking	Best result buried at position #7	Always rerank for production

💡

Practical Takeaways

RAG has two phases — ingestion (offline, batch) and query (online, real-time)

Same embedding model everywhere — query and documents must be in the same vector space

Retrieve more, use less — get 20-50 candidates, rerank to top 5

LLM is the biggest cost and latency — optimize retrieval to minimize what you send to the LLM

snippet
code
Always return sources — let users verify the answer

Start simple, iterate — basic RAG pipeline first, then add hybrid search, reranking, advanced chunking

🔬

What's Next?

Episode 23: Document Parsing — The ingestion pipeline starts with raw documents. PDFs, HTML, scanned images, tables. Getting clean text out of messy formats is a whole engineering challenge. Let's dig into it.

← Previous

Ep 21: Reranking

Next: Episode 23 — Document Parsing

Your boss hands you a 200-page PDF. 'Build a chatbot that answers questions about this.' Step one: extract the text. Harder than it sounds.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.

The Two Phases

PHASE 1: INGESTION (offline, batch)

PHASE 2: QUERY (online, real-time)

Phase 1: Ingestion Pipeline

Step 1: Document Collection

Sources:

Step 2: Document Parsing

Load different formats

Step 3: Chunking

Step 4: Embedding

Convert each chunk into a vector:

Embed in batches (API limits)

Using nomic-embed-text locally

Step 5: Store in Vector Database

Create collection

Upload points

Batch upload

The Complete Ingestion Pipeline

Phase 2: Query Pipeline

Step 1: Embed the Query

Step 2: Retrieve Candidates

Vector search

Step 3: Rerank

Get top 5 reranked chunks

Step 4: Build the Prompt

Context:

Step 5: Generate the Answer

The Complete Query Pipeline

The Full Architecture

Latency Breakdown

Step Time Notes

Step Cost Provider

Common Mistakes

Mistake Impact Fix

Practical Takeaways

Ep 21: Reranking

Ep 23: Document Parsing

Next: Episode 23 — Document Parsing