Document Parsing: Getting Clean Text From Messy Formats

Your boss hands you a 200-page PDF. "Build a chatbot that answers questions about this."

You open the PDF. It has two-column layouts, tables, headers, footers, page numbers, images with captions, footnotes, and a watermark on every page. Some pages are scanned images. The table of contents has dotted leaders. Page 47 is rotated sideways.

Welcome to document parsing — the unglamorous, critical first step of every RAG pipeline. Garbage in, garbage out. If your parser extracts junk, your embeddings encode junk, and your LLM generates confident-sounding junk.

This is the episode nobody wants to write and everybody needs to read.

Why Parsing Is Hard

Text in a PDF isn't stored as you see it. There's no concept of "paragraph" in a PDF — it's a list of instructions like:

BT

/F1 12 Tf % Set font to F1, 12pt

100 700 Td % Move to position (100, 700)

(Hello World) Tj % Draw "Hello World"

ET

The PDF tells the viewer where to draw each character. It doesn't store sentences, paragraphs, or reading order. Multi-column layouts? The PDF doesn't know they're columns — it just has text drawn at specific coordinates.

HTML is better (it has

tags and structure) but still messy — ads, navigation, cookie banners, JavaScript-rendered content.

Scanned documents? You're looking at an image. There is no text. You need OCR (Optical Character Recognition) just to get started.

PDF Extraction: The Options

Option 1: PyPDF (Simple, Fast, Fragile)

python

from pypdf import PdfReader

reader = PdfReader("document.pdf")

for page in reader.pages:

text = page.extract_text()
    print(text)

How it works: Reads the PDF's internal text streams. No AI, no OCR — just extraction.

Good at: Simple PDFs with single-column, left-to-right text. Bad at: Multi-column layouts (text gets interleaved), tables (cells become jumbled), scanned PDFs (returns nothing).

Option 2: pdfplumber (Better Layout Awareness)

python

import pdfplumber

with pdfplumber.open("document.pdf") as pdf:

for page in pdf.pages:
        text = page.extract_text()

# Extract tables separately!
        tables = page.extract_tables()
        for table in tables:
            for row in table:
                print(row)

Advantage: Table extraction. pdfplumber understands cell boundaries and can extract structured table data. It also handles multi-column layouts better by analyzing character positions.

Option 3: Unstructured.io (Multi-Format, Smart)

python

from unstructured.partition.pdf import partition_pdf

elements = partition_pdf(
"document.pdf",
    strategy="hi_res",        # Use layout detection model
    infer_table_structure=True,
    extract_images_in_pdf=True,

)

for element in elements:

print(f"Type: {element.category}")  # Title, NarrativeText, Table, Image
    print(f"Text: {element.text[:100]}")
    print("---")

How it works: Uses an AI layout detection model (detectron2) to identify titles, paragraphs, tables, images, and their reading order. It's not just extracting text — it's understanding the document's structure.

Output types:

Title — headers and section titles

NarrativeText — body paragraphs

Table — extracted as HTML or markdown tables

Image — extracted images with optional captioning

ListItem — bullet points and numbered lists

Option 4: Docling (IBM's Document Parser)

python

from docling.document_converter import DocumentConverter

converter = DocumentConverter()
result = converter.convert("document.pdf")

Get markdown output

markdown = result.document.export_to_markdown()
print(markdown)

How it works: IBM's open-source document parser. Uses transformer models for layout detection. Outputs clean markdown with proper headers, tables, and structure.

Comparison

Tool Tables Scanned PDFs Multi-column Speed Quality

🔧

HTML to Text

Web pages are structured but noisy. You want the article content, not the nav bar, sidebar, ads, and cookie consent popup.

Option 1: BeautifulSoup (Manual)

python

from bs4 import BeautifulSoup

import requests

html = requests.get("https://example.com/article").text
soup = BeautifulSoup(html, "html.parser")

Remove noise

for tag in soup(["script", "style", "nav", "footer", "header", "aside"]):

tag.decompose()

text = soup.get_text(separator="\n", strip=True)

Option 2: Trafilatura (Article Extraction)

python

import trafilatura

url = "https://example.com/article"
downloaded = trafilatura.fetch_url(url)
text = trafilatura.extract(downloaded, include_tables=True, include_links=True)

Trafilatura is specifically designed to extract main content from web pages. It strips navigation, ads, and boilerplate automatically. It's what you want 90% of the time.

Option 3: Jina Reader API

bash

Prefix any URL with r.jina.ai

curl https://r.jina.ai/https://example.com/article

Returns clean markdown. No parsing needed. Free tier available.

📊

OCR: When the Document Is an Image

Scanned PDFs, photos of whiteboards, handwritten notes — these are images, not text.

Tesseract (Free, Open Source)

python

import pytesseract

from PIL import Image

Simple OCR

text = pytesseract.image_to_string(Image.open("scanned_page.png"))

With language specification

text = pytesseract.image_to_string(
Image.open("scanned_page.png"),
    lang="eng+hin"  # English + Hindi

)

EasyOCR (Better for Multiple Languages)

python

import easyocr

reader = easyocr.Reader(['en', 'hi'])  # English + Hindi
results = reader.readtext("scanned_page.png")
for (bbox, text, confidence) in results:
print(f"[{confidence:.2f}] {text}")

Vision LLMs (The Nuclear Option)

For complex documents where traditional OCR fails:

python

from openai import OpenAI

import base64

client = OpenAI()

with open("complex_page.png", "rb") as f:

image_data = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Extract all text from this document image. Preserve structure, tables, and formatting as markdown."},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_data}"}}
        ]
    }]

)

text = response.choices[0].message.content

When to use vision LLMs: Complex layouts, mixed handwriting/print, documents in unusual languages, or when traditional OCR output is too noisy to be useful. Expensive but accurate.

Table Extraction: The Hardest Problem

Tables are where every parser struggles. A simple table:

NameRoleExperience AbhishekEngineer12 years PriyaDesigner8 years

What PyPDF sees: "Name Role Experience Abhishek Engineer 12 years Priya Designer 8 years"

All structure is lost. "12 years" could belong to any column.

Approaches

pdfplumber: Detects cell boundaries using ruled lines. Works well for bordered tables.

python

import pdfplumber

with pdfplumber.open("report.pdf") as pdf:

page = pdf.pages[0]
    tables = page.extract_tables()

for table in tables:
        headers = table[0]
        for row in table[1:]:
            record = dict(zip(headers, row))
            print(record)

Camelot: Specialized PDF table extractor:

python

import camelot

tables = camelot.read_pdf("report.pdf", pages="all")

for table in tables:

df = table.df  # Pandas DataFrame!
    print(df)

Unstructured with hi_res: AI-based table detection:

python

elements = partition_pdf("report.pdf", strategy="hi_res", infer_table_structure=True)
tables = [el for el in elements if el.category == "Table"]

for table in tables:

print(table.metadata.text_as_html)  # HTML table structure

Multimodal Parsing: The Future

The latest approach: skip text extraction entirely. Feed the raw document image to a vision model and let it understand the content.

python

Using LlamaParse (API)

from llama_parse import LlamaParse

parser = LlamaParse(
result_type="markdown",
    use_vendor_multimodal_model=True,
    vendor_multimodal_model_name="openai-gpt-4o",

)

documents = parser.load_data("complex_report.pdf")

for doc in documents:

print(doc.text)  # Clean markdown with tables, structure preserved

Why this works: Vision models can "see" the layout. They understand that a two-column layout should be read column by column. They can read tables by looking at the grid structure. They can handle mixed content (text + charts + diagrams) in one pass.

The trade-off: Slow and expensive. Processing a 50-page PDF through GPT-4o costs several dollars and takes minutes. But the quality is often unmatched.

🔬

The Parsing Decision Tree

What format is your document?

│

├── PDF

│ ├── Simple, text-based → PyPDF or pdfplumber

│ ├── Has tables → pdfplumber or Camelot

│ ├── Complex layout → Unstructured (hi_res) or Docling

│ ├── Scanned/image PDF → OCR first, then extract

│ └── Critical accuracy needed → Vision LLM (GPT-4o)

│

├── HTML/Web page

│ ├── Article content → Trafilatura

│ ├── Full page needed → BeautifulSoup

│ └── Quick and clean → Jina Reader API

│

├── Office docs (Word, Excel, PPT)

│ ├── Word → python-docx or Unstructured

│ ├── Excel → openpyxl or pandas

│ └── PowerPoint → python-pptx

│

└── Images (screenshots, photos)

├── Printed text → Tesseract or EasyOCR
    ├── Handwriting → Vision LLM
    └── Complex (charts, diagrams) → Vision LLM

🛡️

Practical Takeaways

Parsing quality determines RAG quality — invest time here before optimizing anything downstream

PyPDF for simple PDFs, Unstructured for complex ones — know when to upgrade

Tables need special handling — pdfplumber or Camelot for structured extraction

Vision LLMs are the nuclear option — expensive but handle anything

Always inspect parsed output — sample 10-20 documents and manually check extraction quality

Trafilatura for web content — it strips boilerplate better than manual BeautifulSoup

📦

What's Next?

Episode 24: Embedding Models — You've been using embedding models as a black box. Time to open them up. nomic-embed, text-embedding-3, BGE — which to pick, what dimensions mean, and the benchmarks that actually matter.

← Previous

Ep 22: RAG Pipeline End-to-End

Next →

Ep 24: Embedding Models

Next: Episode 24 — Embedding Models

'Just use OpenAI's embedding model.' That's like saying 'just use a Toyota' without knowing if you need a city car or a truck.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.