Chunking Strategies: Why How You Split Documents Changes Everything

You've got a 50-page PDF about Kubernetes. You want to search it with embeddings. But you can't embed the whole document as one vector — the embedding model has a token limit (usually 512-8192 tokens), and even if it didn't, cramming 50 pages into one vector washes out all the specific details.

So you chunk it. Split the document into smaller pieces. Embed each chunk. Store them in your vector database.

Simple, right? Except: chunk too big and your retrieval is vague. Chunk too small and you lose context. Chunk at the wrong boundaries and you split a sentence in half. The chunking strategy you pick is the single biggest lever for RAG quality — more impactful than your embedding model, your vector database, or your LLM.

Why Chunk Size Matters So Much

Imagine searching for "how to set resource limits for Kubernetes pods."

Chunk too large (2000 tokens): The chunk contains resource limits, plus networking, plus storage volumes, plus health checks. The embedding captures the general topic (Kubernetes) but not the specific subtopic (resource limits). Your search returns vaguely relevant chunks.

Chunk too small (50 tokens): The chunk contains "Set the memory limit using resources.limits.memory: 512Mi in the pod spec." Perfect specificity — but zero context. The LLM doesn't know which pod, which deployment, or why.

Chunk just right (200-500 tokens): The chunk covers resource limits comprehensively — CPU and memory limits, requests vs limits, what happens when limits are exceeded — with enough context to be self-explanatory.

This is the chunking Goldilocks problem. And there's no universal "right" answer — it depends on your content and your queries.

Strategy 1: Fixed-Size Chunking

The simplest approach. Split text every N characters or N tokens.

python

def fixed_size_chunks(text, chunk_size=500, overlap=50):
chunks = []
    for i in range(0, len(text), chunk_size - overlap):
        chunk = text[i:i + chunk_size]
        chunks.append(chunk)
    return chunks
text = "Your very long document text here..."
chunks = fixed_size_chunks(text, chunk_size=500, overlap=50)

Pros:

Dead simple to implement

Predictable chunk sizes

Works for any text format

Cons:

Splits mid-sentence, mid-paragraph, mid-thought

No respect for document structure

A chunk might start with "...continued from previous section" — meaningless in isolation

When to use: Quick prototyping, unstructured text with no clear section boundaries.

🔧

Strategy 2: Recursive Character Splitting

The LangChain default. Try to split on natural boundaries, falling back to smaller separators:

python

LangChain's RecursiveCharacterTextSplitter logic:

separators = [
"\n\n",    # Double newline (paragraph breaks) — try first
    "\n",      # Single newline (line breaks)
    ". ",      # Sentence endings
    " ",       # Word boundaries
    ""         # Individual characters (last resort)

]

Algorithm:

1. Try splitting on "\n\n" (paragraphs)

2. If chunks are still too big, split on "\n"

3. If still too big, split on ". "

4. Keep going until chunks fit under the limit

python

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
chunk_size=500,
    chunk_overlap=50,
    separators=["\n\n", "\n", ". ", " ", ""]

)

chunks = splitter.split_text(document_text)

Pros:

Respects natural text boundaries

Much better than fixed-size for most content

Highly configurable separators

Cons:

Still purely character-based — doesn't understand meaning

Can still split related content across chunks

When to use: The safe default for most RAG applications. Start here.

📊

Strategy 3: Semantic Chunking

Split based on meaning changes, not character counts. Use embeddings to detect when the topic shifts.

python

from langchain_experimental.text_splitter import SemanticChunker

from langchain_openai import OpenAIEmbeddings

splitter = SemanticChunker(
OpenAIEmbeddings(),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95

)

chunks = splitter.split_text(document_text)

How it works:

Sentence 1: "Kubernetes uses pods as the smallest deployable unit."

Sentence 2: "Each pod can contain one or more containers."

Sentence 3: "Pods share networking and storage."

← Similarity between sentences 3 and 4: 0.42 (LOW — topic change!) →

Sentence 4: "To deploy a service, you need to create a Service resource."

Sentence 5: "Services provide stable endpoints for pods."

Result:

Chunk 1: Sentences 1-3 (about pods)

Chunk 2: Sentences 4-5 (about services)

Pros:

Chunks align with actual topic boundaries

Variable-length chunks that match content structure

Semantically coherent chunks = better retrieval

Cons:

Requires running embeddings on every sentence (expensive during ingestion)

Slower ingestion pipeline

Can produce very uneven chunk sizes

When to use: High-quality RAG where retrieval precision matters more than ingestion speed.

Strategy 4: Document Structure-Based

Use the document's own structure — headers, sections, code blocks — as chunk boundaries.

python

For Markdown

from langchain.text_splitter import MarkdownHeaderTextSplitter

headers_to_split = [
("#", "h1"),
    ("##", "h2"),
    ("###", "h3"),

]

splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split)
chunks = splitter.split_text(markdown_text)

Each chunk includes its header hierarchy as metadata:

{ "h1": "Kubernetes Guide", "h2": "Pod Configuration", "content": "..." }

For HTML:

python

from langchain.text_splitter import HTMLHeaderTextSplitter

splitter = HTMLHeaderTextSplitter(
headers_to_split_on=[("h1", "h1"), ("h2", "h2"), ("h3", "h3")]

)

chunks = splitter.split_text(html_content)

Pros:

Leverages human-authored structure

Headers become metadata (great for filtering)

Most natural chunking for documentation

Cons:

Only works for well-structured documents

Sections can still be too large (need secondary splitting)

Doesn't work for unstructured text

When to use: Documentation, technical manuals, knowledge bases — anything with headers.

The Overlap Problem

Without overlap, context is lost at chunk boundaries:

Chunk 1: "...Kubernetes resource limits define the maximum CPU and memory

a container can use. The default namespace has"

Chunk 2: "no resource quotas applied. You should always set limits

in production to prevent noisy neighbor problems..."

The critical sentence "The default namespace has no resource quotas applied" is split across two chunks. Neither chunk contains the full thought.

Overlap fixes this:

Chunk 1: "...resource limits define the maximum CPU and memory

a container can use. The default namespace has
          no resource quotas applied."
                                    ↑ overlap zone

Chunk 2: "The default namespace has no resource quotas applied.

You should always set limits in production to prevent
          noisy neighbor problems..."
          ↑ overlap zone starts here

How Much Overlap?

Overlap Effect

Rule of thumb: 10-15% of chunk size. For a 500-token chunk, use 50-75 tokens of overlap.

🔬

Chunk Size: The Numbers

Based on common practice and benchmarks:

Chunk Size Best For Trade-off

The empirical winner for most RAG: 256-512 tokens with 10-15% overlap. But always test with YOUR data and YOUR queries.

python

Quick experiment to find optimal chunk size

chunk_sizes = [128, 256, 512, 1024]

for size in chunk_sizes:

chunks = split_document(doc, chunk_size=size, overlap=int(size * 0.1))
    results = search(query, chunks)
    relevance = evaluate_results(results, ground_truth)
    print(f"Size: {size:4d}  Chunks: {len(chunks):4d}  Relevance: {relevance:.3f}")

🛡️

Advanced: Multi-Level Chunking

The best RAG systems don't use just one chunk size. They use parent-child chunking:

Parent chunk (1000 tokens):

"This section covers Kubernetes resource management.

Resource limits define the maximum CPU and memory...

Resource requests define the minimum guaranteed...

LimitRange objects set defaults for a namespace..."

Child chunks (200 tokens each):

Child 1: "Resource limits define the maximum CPU and memory..."

Child 2: "Resource requests define the minimum guaranteed..."

Child 3: "LimitRange objects set defaults for a namespace..."

Search against child chunks (specific), but return parent chunks (contextual). Best of both worlds. We'll cover this in detail in Episode 25: Retrieval Strategies.

📦

Practical Takeaways

Chunking is the #1 lever for RAG quality — spend more time here than on model selection

Start with RecursiveCharacterTextSplitter — it's the safe default (256-512 tokens, 10% overlap)

Use document structure when available — Markdown/HTML headers make natural chunk boundaries

Always use overlap — 10-15% prevents losing context at boundaries

Test chunk sizes with your actual queries — there's no universal optimum

Consider semantic chunking for high-value data — more expensive but better retrieval

🚀

What's Next?

Episode 20: Hybrid Search — Vector search finds semantically similar results, but sometimes you need exact keyword matches. "Error code KB-4092" won't work with cosine similarity. Hybrid search combines the best of both worlds.

← Previous

Ep 18: Vector Databases

Next →

Ep 20: Hybrid Search

Next: Episode 20 — Hybrid Search

User searches 'error KB-4092'. Vector search returns semantically relevant results. None mention the actual error code.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.