Reranking: Why Your First Search Results Aren't the Best

You search "how to handle database connection timeouts in Python." Your vector DB returns 10 results. Result #1 is about Python exception handling. Result #7 is a perfect guide on psycopg2 connection pool timeout configuration.

Why is #7 not #1?

Because your embedding model used a bi-encoder — it embedded the query and the documents separately, then compared their vectors. Fast, but it can't see the nuances. It doesn't know that "database connection timeouts" specifically means psycopg2 pool configuration in this context.

A reranker takes those 10 results and rescores them by reading the query AND each document together, word by word. It sees the nuances. It pushes #7 to #1.

Two-stage retrieval: cast a wide net, then pick the best fish. This is how every serious search system works — Google, Bing, Amazon, Netflix. And now your RAG pipeline.

Bi-Encoders vs Cross-Encoders

This is the fundamental trade-off behind all of search.

Bi-Encoder (What You've Been Using)

The embedding model you've been using is a bi-encoder. It encodes the query and documents independently:

Query: "database timeout Python" → Encoder → [0.23, -0.41, 0.87, ...]

↓
                                              Cosine Similarity → 0.82
                                                     ↑

Doc: "Handling psycopg2 pool..." → Encoder → [0.19, -0.38, 0.91, ...]

Key property: The document embedding is computed ONCE during indexing. At query time, you only embed the query and compute similarities. This is why it scales — you can search millions of documents in milliseconds.

The limitation: The query and document never "see" each other. The encoder can't reason about how they interact. It's like judging compatibility from two separate resumes without ever putting the people in a room together.

Cross-Encoder (The Reranker)

A cross-encoder reads the query and document TOGETHER:

Input: "[CLS] database timeout Python [SEP] Handling psycopg2 connection

pool timeout configuration with retry logic in Python [SEP]"
                              ↓
                      Transformer (all tokens attend to all tokens)
                              ↓
                      Relevance Score: 0.94

Key property: Every word in the query attends to every word in the document. The model sees that "timeout" in the query relates to "timeout configuration" in the document, that "database" relates to "psycopg2" (a database driver), and that "Python" appears in both.

The limitation: You can't pre-compute anything. For each query, you must run the cross-encoder on EVERY candidate document. With 10 million documents, that's 10 million forward passes — way too slow.

The Comparison

Bi-Encoder Cross-Encoder

The solution is obvious: use the bi-encoder to get top 50-100 candidates fast, then use the cross-encoder to rerank those candidates accurately.

Two-Stage Retrieval Pipeline

User Query

↓

Stage 1: RETRIEVAL (bi-encoder + vector DB)

"Get me the top 50 candidates"
    Speed: 5-20ms for millions of documents
    Recall: ~90% (might miss some relevant docs)
    ↓

50 candidates

↓

Stage 2: RERANKING (cross-encoder)

"Score each of these 50 against the query"
    Speed: 50 × 10ms = 500ms
    Precision: ~95%+ (much better ordering)
    ↓

Top 5 results → LLM context

Without reranking, you send the top 5 from stage 1 — which might have the best result at position #7. With reranking, the cross-encoder fixes the ordering and pushes the actually-best result to #1.

🔧

Reranking Models

Cohere Rerank

The most popular API reranker. Dead simple:

python

import cohere

co = cohere.Client("YOUR_API_KEY")
results = co.rerank(
model="rerank-english-v3.0",
    query="database timeout Python",
    documents=[
        "Handling psycopg2 connection pool timeout configuration...",
        "Python exception handling best practices...",
        "Database migration strategies in Django...",
        "Connection retry logic with exponential backoff...",
    ],
    top_n=3,
    return_documents=True

)

for r in results.results:

print(f"Score: {r.relevance_score:.4f}  Doc: {r.document.text[:60]}")

Score: 0.9842 Doc: Handling psycopg2 connection pool timeout configuration...

Score: 0.8231 Doc: Connection retry logic with exponential backoff...

Score: 0.4102 Doc: Python exception handling best practices...

Pricing: ~$0.002 per 1000 documents reranked. Cheap enough for production.

Open Source: Cross-Encoder Models

python

from sentence_transformers import CrossEncoder

model = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
query = "database timeout Python"
documents = [
"Handling psycopg2 connection pool timeout configuration...",
    "Python exception handling best practices...",
    "Database migration strategies in Django...",

]

Score each query-document pair

pairs = [[query, doc] for doc in documents]
scores = model.predict(pairs)

Sort by score

ranked = sorted(zip(documents, scores), key=lambda x: x[1], reverse=True)

for doc, score in ranked:

print(f"Score: {score:.4f}  Doc: {doc[:60]}")

Popular Reranker Models

Model Type Speed Quality Best For

📊

The Impact: Before vs After Reranking

Real benchmark from a RAG evaluation on a technical documentation corpus:

Metric Without Reranking With Reranking

─────────────────────────────────────────────────────────────

Recall@5 0.72 0.72 (same — reranking doesn't add docs)

Precision@5 0.58 0.81 (+40%)

MRR (Mean Reciprocal 0.65 0.88 (+35%)

Rank)

NDCG@5 0.61 0.84 (+38%)

Same retrieved documents, dramatically better ordering. The best result is now actually on top.

What This Means for RAG

If you retrieve top-5 chunks and stuff them into the LLM prompt:

Without reranking: The most relevant chunk might be at position #4 (or not in top-5 at all from the retrieval pool)

With reranking: The most relevant chunk is at position #1

Since LLMs are affected by the "lost in the middle" problem (more on this in Episode 26), having the best content at position #1 directly improves answer quality.

When NOT to Rerank

Reranking isn't free. Here's when to skip it:

Situation Why Skip Reranking

Simple FAQ lookup Bi-encoder retrieval is already precise

Rule of thumb: If your retrieval precision is already above 85% and latency matters, skip reranking. If precision is below 75%, reranking is the highest-impact improvement you can make.

Full Pipeline Integration

Here's a complete two-stage retrieval with Qdrant + Cohere reranking:

python

import cohere

from qdrant_client import QdrantClient

Initialize

qdrant = QdrantClient("localhost", port=6333)
co = cohere.Client("YOUR_COHERE_KEY")
def search_with_reranking(query: str, top_k: int = 5):
# Stage 1: Vector retrieval (wide net)
    query_vector = embed(query)  # Your embedding function
    candidates = qdrant.search(
        collection_name="docs",
        query_vector=query_vector,
        limit=50,  # Retrieve 50 candidates
    )
# Stage 2: Rerank (pick the best)
    docs = [hit.payload["content"] for hit in candidates]
    reranked = co.rerank(
        model="rerank-english-v3.0",
        query=query,
        documents=docs,
        top_n=top_k,
    )
# Return reranked results with original metadata
    results = []
    for r in reranked.results:
        original = candidates[r.index]
        results.append({
            "content": original.payload["content"],
            "metadata": original.payload,
            "rerank_score": r.relevance_score,
            "vector_score": original.score,
        })
return results

🔬

Practical Takeaways

Reranking is the biggest quick win for RAG quality — 30-40% improvement in precision

Two-stage pipeline: bi-encoder retrieves 50-100 candidates, cross-encoder reranks top-K

Cross-encoders are more accurate but can't scale — that's why you need both stages

Cohere Rerank is the easiest start — API, no infrastructure, cheap

Self-host with bge-reranker if you need privacy or can't use external APIs

Retrieve more, rerank fewer — get 50 candidates, rerank to top 5

🛡️

What's Next?

Episode 22: RAG Pipeline End-to-End — You've learned embeddings, vector search, chunking, hybrid search, and reranking. Now let's put it all together into a complete pipeline: ingestion → chunking → embedding → storage → retrieval → generation. The full picture.

← Previous

Ep 20: Hybrid Search

Next →

Ep 22: RAG Pipeline End-to-End

Next: Episode 22 — RAG Pipeline End-to-End

You've learned the pieces: embeddings, vector databases, chunking, hybrid search, reranking. Now let's build the whole machine.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.