Context Stuffing: How to Feed Documents to an LLM Without Losing the Plot

You spent weeks building a retrieval pipeline. Your chunks are perfectly sized, your reranker pushes the best results to the top, your vector database is tuned. You retrieve 5 beautiful, relevant chunks.

Then you jam them into a prompt, the LLM ignores chunk #3 (which had the actual answer), and generates a hallucinated response from chunk #1 (which was tangentially related).

Retrieval is only half the battle. How you present the retrieved context to the LLM determines whether it actually uses it. This is context stuffing — and it's more nuanced than most people realize.

The Lost-in-the-Middle Problem

In 2023, Stanford researchers discovered something alarming: LLMs pay the most attention to information at the beginning and end of their context, and tend to ignore what's in the middle.

Attention Distribution:

████████░░░░░░░░░░░░████████

↑ Start End ↑

(high attention) (high attention)

░░░░░░░░████████████░░░░░░░░

↑ Middle ↑
          (LOW attention!)

This was tested with GPT-3.5, GPT-4, and Claude. When the relevant information was placed at positions 1 or 10 (out of 10 documents), the model answered correctly 70-80% of the time. When placed at position 5 (the middle), accuracy dropped to 40-50%.

Let that sink in. Your model is twice as likely to miss the answer if it's in the middle of the context.

What This Means for RAG

If you retrieve 5 chunks and stuff them in order:

[Chunk 1 - somewhat relevant] ← LLM pays attention here

[Chunk 2 - somewhat relevant]

[Chunk 3 - PERFECT ANSWER] ← LLM ignores this

[Chunk 4 - somewhat relevant]

[Chunk 5 - somewhat relevant] ← LLM pays attention here

The model generates an answer from chunks 1 and 5, missing the actual answer in chunk 3.

Ordering Strategies

Strategy 1: Best-First

Put the most relevant chunk first.

python

def best_first_ordering(chunks_with_scores):

# Already sorted by reranker score (highest first)
    return [chunk for chunk, score in chunks_with_scores]

Pros: The most relevant content gets maximum attention. Cons: If the best chunk doesn't fully answer the question, the model fixates on it and ignores supplementary info in the middle.

Strategy 2: Best-at-Edges

Place the most relevant chunks at the beginning AND end. Put least relevant in the middle.

python

def edges_ordering(chunks_with_scores):

sorted_chunks = [c for c, s in sorted(chunks_with_scores, key=lambda x: x[1], reverse=True)]

result = []
    for i, chunk in enumerate(sorted_chunks):
        if i % 2 == 0:
            result.insert(0, chunk)  # Add to beginning
        else:
            result.append(chunk)     # Add to end

return result

Input order (by relevance): [Best, Good, OK, Meh, Worst]

Output order: [OK, Good, Best, Worst, Meh]

↑ edges = high attention ↑
                                 ↑ middle = low ↑

Wait, that doesn't look right. Let me rethink:

python

def edges_ordering(chunks_with_scores):

"""Place best chunks at position 1 and last position."""
    ranked = sorted(chunks_with_scores, key=lambda x: x[1], reverse=True)

if len(ranked) <= 2:
        return [c for c, s in ranked]

# Best at position 1, second-best at last position
    result = [ranked[0][0]]                    # Best → first
    middle = [c for c, s in ranked[2:]]        # Rest → middle
    result.extend(middle)
    result.append(ranked[1][0])                # 2nd best → last

return result

This is the recommended approach for most RAG systems.

Strategy 3: Reverse Order (Worst-First)

Some practitioners find that putting the least relevant first and best last works well — the model "builds up" to the answer:

python

def reverse_ordering(chunks_with_scores):

return [c for c, s in sorted(chunks_with_scores, key=lambda x: x[1])]
    # Worst first → Best last (recency bias helps)

🔧

Prompt Templates That Work

The Basic Template

python

BASIC_RAG_PROMPT = """Answer the user's question based ONLY on the following context.

If the context doesn't contain enough information to answer, say "I don't have enough information."

Context:

{context}

Question: {question}

Answer:"""

The Structured Template (Better)

python

STRUCTURED_RAG_PROMPT = """You are a helpful assistant that answers questions based on provided documents.

Instructions

1. Read ALL provided documents carefully
2. Answer ONLY based on information in the documents
3. If documents conflict, mention both perspectives
4. If information is insufficient, say so explicitly
5. Cite which document(s) support your answer

Documents

{numbered_documents}

Question

{question}

Answer (cite document numbers)"""

def format_documents(chunks):

numbered = []
    for i, chunk in enumerate(chunks, 1):
        numbered.append(f"[Document {i}]\nSource: {chunk['source']}\n{chunk['content']}\n")
    return "\n".join(numbered)

The Citation-Enforcing Template

python

CITATION_PROMPT = """Answer the question using ONLY the provided sources.

Every claim must include a citation in [Source X] format.

Sources:

{sources}

Question: {question}

Provide a detailed answer with citations:"""

Forcing citations does two things:

Makes the model ground its answer in the actual context

Makes hallucinations detectable (if it cites [Source 3] but source 3 doesn't support the claim)

📊

How Many Chunks to Stuff?

More context ≠ better answers. There's a sweet spot.

Chunks Effect When To Use

The research says 3-5 chunks is optimal for most question-answering tasks. Beyond that, you're adding noise faster than signal.

python

Dynamic chunk count based on query complexity

def determine_chunk_count(query: str) -> int:
# Simple heuristic: longer/complex queries need more context
    word_count = len(query.split())
if word_count < 10:
        return 3  # Simple question
    elif word_count < 25:
        return 5  # Moderate question
    else:
        return 8  # Complex question

Context Compression

Instead of stuffing raw chunks, compress them first:

python

def compress_context(query: str, chunks: list[str]) -> str:
"""Use an LLM to extract only query-relevant information from each chunk."""
    compressed = []
    for chunk in chunks:
        prompt = f"""Extract ONLY the information relevant to answering this question.
        Remove everything else. Be concise.
Question: {query}
Document:
        {chunk}
Relevant information:"""
relevant_info = llm.generate(prompt, model="gpt-4o-mini")  # Cheap model
        if relevant_info.strip() and relevant_info != "No relevant information found.":
            compressed.append(relevant_info)
return "\n\n".join(compressed)

Trade-off: Extra LLM call per chunk, but dramatically reduces the context the main LLM needs to process. Especially useful when chunks are large (500+ tokens) but only a sentence or two is relevant.

System Prompt Engineering for RAG

The system prompt shapes how the LLM interacts with context:

python

SYSTEM_PROMPT = """You are a precise, helpful assistant for {company_name}.

Core Rules

• Base ALL answers on the provided documents
• NEVER use prior knowledge to fill gaps
• If the documents don't cover the topic, say: "Based on the available documents, I cannot answer this question."
• When documents provide partial information, clearly state what IS covered and what ISN'T

Citation Style

• Reference documents as [Doc 1], [Doc 2], etc.
• Place citations immediately after the supported claim
• If multiple documents support a claim, cite all of them

Handling Conflicts

• If documents contradict each other, present both perspectives
• Note which document is more recent if dates are available
• Never silently pick one perspective over another

Formatting

• Use bullet points for lists
• Use headers for long answers
• Keep responses concise but complete"""

🔬

The Token Budget

Every token in your context costs money and competes for attention. Budget wisely:

Total context budget: 4000 tokens (example)

─────────────────────────────────────────

System prompt: ~200 tokens

Retrieved chunks: ~3000 tokens (3-5 chunks × 500-700 tokens)

User question: ~50 tokens

Response instruction: ~50 tokens

─────────────────────────────────────────

Leaves ~700 tokens for other instructions or conversation history

For multi-turn conversations, you also need to budget for chat history:

python

def build_rag_prompt(query, chunks, chat_history, max_context_tokens=4000):
system_tokens = 200
    query_tokens = count_tokens(query)
    history_tokens = count_tokens(format_history(chat_history[-3:]))  # Last 3 turns
available_for_chunks = max_context_tokens - system_tokens - query_tokens - history_tokens
# Fill with as many chunks as fit
    selected_chunks = []
    current_tokens = 0
    for chunk in chunks:
        chunk_tokens = count_tokens(chunk)
        if current_tokens + chunk_tokens > available_for_chunks:
            break
        selected_chunks.append(chunk)
        current_tokens += chunk_tokens
return build_prompt(selected_chunks, query, chat_history[-3:])

🛡️

Anti-Patterns to Avoid

Anti-Pattern Why It's Bad Fix

📦

Practical Takeaways

Lost-in-the-middle is real — put best chunks at the beginning and end of context

3-5 chunks is the sweet spot — more context adds noise, not signal

Force citations — makes grounding visible and hallucinations detectable

Explicit "use only context" instructions — without them, the LLM will use prior knowledge

Budget your tokens — system prompt + history + chunks + question must all fit

Consider context compression — extract only relevant info from each chunk before stuffing

🚀

What's Next?

Episode 27: RAG Evaluation — You've built the pipeline. How do you know if it's actually good? Faithfulness, relevance, recall — the RAGAS framework for measuring RAG quality, plus when human eval beats automated metrics.

← Previous

Ep 25: Retrieval Strategies

Next →

Ep 27: RAG Evaluation

Next: Episode 27 — RAG Evaluation

Your RAG pipeline is live. Users are asking questions. But is it answering correctly? How would you even know?

This is part of a 98-episode series covering AI engineering from tokens to production deployment.