Embedding Models: Which One to Pick and Why It Matters

"Just use OpenAI's embedding model."

That's the advice you'll hear everywhere. And it's... fine. But it's like saying "just use a Toyota" without knowing if you need a city car or a truck. The embedding model you pick affects retrieval quality, latency, cost, and whether you can run things offline.

There are now dozens of embedding models — some run locally on your laptop, some need GPU clusters, some are free, some charge per million tokens. The wrong choice costs you quality and money. The right choice can be free and better.

Let's open up the field and actually compare them.

What Makes an Embedding Model "Good"

An embedding model is evaluated on how well its vectors capture semantic relationships. The key metrics:

MTEB Benchmark (Massive Text Embedding Benchmark)

MTEB is the standard benchmark suite. It tests embedding models across multiple tasks:

Task What It Tests

For RAG, the Retrieval score is what you care about most. A model that's great at clustering but mediocre at retrieval won't help your search pipeline.

Check the leaderboard: huggingface.co/spaces/mteb/leaderboard

The Big Players

OpenAI text-embedding-3-small

python

from openai import OpenAI

client = OpenAI()
response = client.embeddings.create(
model="text-embedding-3-small",
    input="Your text here"

)

vector = response.data[0].embedding  # 1536 dimensions

Spec Value

The workhorse. Good enough for most applications. The cost is negligible — embedding your entire Wikipedia corpus costs under $10.

OpenAI text-embedding-3-large

python

response = client.embeddings.create(
model="text-embedding-3-large",
    input="Your text here",
    dimensions=3072  # Full dimensions (or reduce: 256, 512, 1024, 1536)

)

Spec Value

The upgrade. ~4% better retrieval than small, 6.5x more expensive. The dimension reduction feature is unique — you can output 256-dim vectors (much less storage) with graceful quality degradation.

nomic-embed-text (Open Source, Local)

bash

Via Ollama (free, local)

ollama pull nomic-embed-text

curl http://localhost:11434/api/embeddings -d '{

"model": "nomic-embed-text",

"prompt": "search_document: Your text here"

}'

Spec Value

My recommendation for getting started. Free, runs locally, competitive with OpenAI small, no API dependency. Uses task prefixes — prepend search_document: for documents and search_query: for queries.

BAAI/bge-large-en-v1.5

python

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('BAAI/bge-large-en-v1.5')
embeddings = model.encode(
["Passage: Your text here"],  # Add "Passage: " prefix for documents
    normalize_embeddings=True

)

Spec Value

Strong open-source option. Part of the BGE (Beijing Academy of AI) series. The 512 token limit is restrictive — your chunks must be small. Use the bge-m3 variant for longer contexts.

BAAI/bge-m3 (Multilingual Beast)

python

from FlagEmbedding import BGEM3FlagModel

model = BGEM3FlagModel('BAAI/bge-m3', use_fp16=True)
embeddings = model.encode(
["Your text in any language"],
    max_length=8192,

)

Returns both dense AND sparse vectors!

dense = embeddings['dense_vecs']
sparse = embeddings['lexical_weights']

Spec Value

The multilingual champion. Supports 100+ languages, long context (8192 tokens), and outputs BOTH dense and sparse vectors — perfect for hybrid search in a single model.

Cohere embed-v3

python

import cohere

co = cohere.Client("YOUR_KEY")
response = co.embed(
texts=["Your text here"],
    model="embed-english-v3.0",
    input_type="search_document",  # or "search_query"

)

Spec Value

API option with input types. Cohere differentiates between document and query embeddings via input_type, which can improve retrieval quality.

🔧

The Comparison Table

📊

Dimensions: Does Size Matter?

More dimensions = more storage + more compute. But do they improve quality?

Dimensions Storage per 1M vectors Retrieval Quality

───────────────────────────────────────────────────────

256 ~1 GB ~90% of max

512 ~2 GB ~95% of max

768 ~3 GB ~97% of max

1024 ~4 GB ~99% of max

1536 ~6 GB ~99.5% of max

3072 ~12 GB 100% (baseline)

The sweet spot is 768-1024 dimensions. You get 97-99% of the quality at half the storage of 1536-dim models. The last 1% of quality rarely justifies doubling your storage and compute costs.

OpenAI's text-embedding-3-large lets you choose dimensions at query time — try 512 or 1024 before defaulting to the full 3072.

Task Prefixes: The Hidden Feature

Several models use task prefixes to improve quality. The model behaves differently for queries vs documents:

Model Document Prefix Query Prefix

This actually matters. Tests show 2-5% retrieval improvement when using correct prefixes vs no prefixes.

python

WRONG — same embedding for query and document

embed("How to deploy ML models")

RIGHT — different treatment

embed("search_query: How to deploy ML models")        # For the query
embed("search_document: This guide covers ML deployment...")  # For the doc

How to Choose: Decision Tree

Are you prototyping?

├── Yes → nomic-embed-text (free, local, good enough)

└── No — production use

│
    ├── Need to run locally / air-gapped?
    │   ├── English only → nomic-embed-text or bge-large
    │   ├── Multilingual → bge-m3 (needs GPU)
    │   └── Long documents → nomic-embed-text (8192 tokens)
    │
    ├── Budget is not a concern?
    │   ├── English → text-embedding-3-large (3072d)
    │   └── Multilingual → Jina embeddings-v3 or bge-m3
    │
    └── Want lowest cost?
        └── text-embedding-3-small ($0.02/1M tokens)

My recommendation ladder:

Start: nomic-embed-text (free, local, 768d)

Scale: text-embedding-3-small (cheap, reliable API)

Optimize: bge-m3 (if you need multilingual or hybrid search)

Max quality: text-embedding-3-large at 1024d (good balance)

🔬

Migration Warning: You Can't Switch Models Easily

Once you embed your documents with Model A, you can't query them with Model B. The vector spaces are incompatible.

Model A's vector space Model B's vector space

• "kubernetes"                  • "kubernetes"
    • "docker"                              • "docker"

• "containers" • "containers"

(their arrangement)              (completely different!)

Switching models means re-embedding your entire corpus. For 10 million documents at $0.02/1M tokens, that's ~$20 and a few hours. Not expensive, but plan for it.

Best practice: Record which model and version you used in your vector DB metadata. When you upgrade models, you'll know what needs re-embedding.

🛡️

Practical Takeaways

nomic-embed-text for local, text-embedding-3-small for API — these cover 90% of use cases

768-1024 dimensions is the sweet spot — don't pay for 3072 unless you've proven you need it

Use task prefixes — 2-5% free quality improvement

Switching models = re-embedding everything — choose carefully, document your choice

MTEB Retrieval score matters most for RAG — ignore other task scores

Test with YOUR data — benchmark scores are averages; your domain might prefer a different model

📦

What's Next?

Episode 25: Retrieval Strategies — Naive RAG (embed → search → generate) is just the beginning. Multi-query retrieval, HyDE, parent-child chunks, contextual retrieval — advanced techniques that push RAG quality from "good" to "production-ready."

← Previous

Ep 23: Document Parsing

Next →

Ep 25: Retrieval Strategies

Next: Episode 25 — Retrieval Strategies

The basic RAG pipeline works surprisingly well. Until it doesn't. Here's how to go beyond naive retrieval.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.