Vector Databases: Where Embeddings Live and How They Work

HNSW gives you fast search. Embeddings give you meaning. But you can't just dump 50 million vectors into a Python list and call it a day.

You need persistence (don't lose vectors on restart). You need filtering ("find similar documents, but only from 2024"). You need scale (one machine won't hold 100 million vectors). You need an API (your app needs to talk to it).

You need a vector database. And right now, there are too many to choose from.

Let's open them up and look inside.

What's Inside a Vector Database

Every vector database, regardless of brand, has these components:

┌─────────────────────────────────────────┐

│ Vector Database │

├─────────────────────────────────────────┤

│ API Layer (REST / gRPC / SDK) │

├─────────────────────────────────────────┤

│ Query Engine │

│ ├── Vector Search (HNSW, IVF, etc.) │

│ ├── Metadata Filtering │

│ └── Hybrid Search (vector + keyword) │

├─────────────────────────────────────────┤

│ Storage Engine │

│ ├── Vector Storage (mmap / RAM) │

│ ├── Payload Storage (metadata) │

│ └── Index Storage (HNSW graph) │

├─────────────────────────────────────────┤

│ Distributed Layer (optional) │

│ ├── Sharding │

│ ├── Replication │

│ └── Consensus │

└─────────────────────────────────────────┘

The Data Model

Every vector DB stores points (or records). Each point has:

json

{

"id": "doc_42",

"vector": [0.023, -0.891, 0.041, ...], // 768 or 1536 floats

"payload": { // metadata

"title": "Deploying ML on Kubernetes",
    "author": "Abhishek",
    "date": "2024-06-15",
    "category": "infrastructure",
    "word_count": 2400

}

}

You search by vector (semantic similarity) and filter by payload (metadata conditions). This combination is what makes vector databases useful for real applications.

The Big Players: Head-to-Head

Qdrant

Written in Rust. Built for production.

bash

Start with Docker

docker run -p 6333:6333 -p 6334:6334 qdrant/qdrant

Create collection

curl -X PUT http://localhost:6333/collections/my_docs \

-H 'Content-Type: application/json' \

-d '{

"vectors": { "size": 768, "distance": "Cosine" },
    "optimizers_config": { "default_segment_number": 2 }

}'

Strengths: Fast, memory-efficient (quantization built-in), excellent filtering, open source, self-hostable. Rust performance means low latency under load.

Payload filtering happens at the HNSW level — not after retrieval. This matters. Some databases fetch K nearest vectors THEN filter, which can return fewer results than requested. Qdrant filters during traversal.

Pinecone

Fully managed. You don't run anything.

python

import pinecone

pc = pinecone.Pinecone(api_key="YOUR_KEY")
index = pc.Index("my-index")

Upsert

index.upsert(vectors=[

{"id": "doc1", "values": [0.1, 0.2, ...], "metadata": {"category": "tech"}}

])

Query

results = index.query(vector=[0.15, 0.22, ...], top_k=10,
filter={"category": {"$eq": "tech"}})

Strengths: Zero ops, scales automatically, serverless option (pay per query). Great for teams that don't want to manage infrastructure.

Weaknesses: Vendor lock-in, can get expensive at scale, no self-hosting option, limited control over indexing parameters.

Weaviate

GraphQL-native, module ecosystem.

graphql

{

Get {

Document(
      nearText: { concepts: ["machine learning deployment"] }
      where: { path: ["category"], operator: Equal, valueText: "infrastructure" }
      limit: 10
    ) {
      title
      content
      _additional { distance }
    }

}

}

Strengths: Built-in vectorization (plug in your embedding model), GraphQL API, good for complex data models, modules for different embedding providers.

Weaknesses: GraphQL can be overkill, Java memory overhead, steeper learning curve.

Chroma

The SQLite of vector databases.

python

import chromadb

client = chromadb.Client()
collection = client.create_collection("my_docs")

Add documents (Chroma embeds them for you!)

collection.add(

documents=["Kubernetes deployment guide", "Python web scraping"],
    metadatas=[{"topic": "infra"}, {"topic": "python"}],
    ids=["doc1", "doc2"]

)

Query

results = collection.query(
query_texts=["how to deploy ML models"],
    n_results=5

)

Strengths: Dead simple API, embeds text for you (built-in embedding), runs in-process (no server needed), perfect for prototyping and small projects.

Weaknesses: Not built for production scale, limited persistence options, single-machine only.

pgvector (PostgreSQL Extension)

Vectors inside your existing Postgres.

sql

-- Enable extension

CREATE EXTENSION vector;

-- Create table

CREATE TABLE documents (

id SERIAL PRIMARY KEY,

content TEXT,

embedding vector(768),

category TEXT,

created_at TIMESTAMP

);

-- Create HNSW index

CREATE INDEX ON documents

USING hnsw (embedding vector_cosine_ops)

WITH (m = 16, ef_construction = 200);

-- Search

SELECT content, 1 - (embedding <=> '[0.1, 0.2, ...]'::vector) as similarity

FROM documents

WHERE category = 'infrastructure'

ORDER BY embedding <=> '[0.1, 0.2, ...]'::vector

LIMIT 10;

Strengths: No new infrastructure — use your existing Postgres, full SQL power, ACID transactions, joins with other tables.

Weaknesses: Slower than purpose-built vector DBs, HNSW support is newer, scaling is Postgres scaling (hard).

🔧

Comparison Table

Feature Qdrant Pinecone Weaviate Chroma pgvector

📊

Indexing: What Happens When You Insert a Vector

When you add a vector, the database doesn't just append it to a file. Here's the pipeline:

New Vector Arrives

↓

1. Validate dimensions (must match collection config)

↓

1. Store vector data (mmap file or RAM)

↓

1. Store payload/metadata (key-value store)

↓

1. Add to HNSW graph

• Find insertion layer (random, exponential distribution)
• Connect to nearest neighbors in each layer
• Prune excessive connections (max M connections)

↓

1. Update payload indexes (if indexed fields)

↓

1. Write to WAL (write-ahead log) for durability

↓

Done. Searchable immediately (or after segment merge)

Segments and Compaction

Most vector databases use segments — chunks of the index:

New vectors go into a mutable segment (optimized for writes)

Periodically, segments are merged and optimized (compaction)

Optimized segments are immutable (optimized for reads)

This is similar to how LSM-trees work in databases like RocksDB. It lets you handle writes efficiently while maintaining fast reads.

Metadata Filtering: The Hidden Complexity

"Find me vectors similar to X where category = 'tech' and date > 2024-01-01."

This sounds simple. It's actually the hardest problem in vector databases. Two approaches:

Pre-filtering

Filter FIRST, then search only within the filtered set.

All vectors (10M) → Filter by metadata (500K match) → HNSW search on 500K

Problem: If the filter is very selective (only 100 vectors match), you can't use HNSW effectively. You end up brute-forcing.

Post-filtering

Search FIRST, get top-K results, then filter.

All vectors (10M) → HNSW top-100 → Filter by metadata → Maybe 3 results left

Problem: You asked for top-10 but only 3 passed the filter. You need to search again with higher K. Wasteful and unpredictable.

The Hybrid Approach (Qdrant's Solution)

Qdrant integrates filtering INTO the HNSW traversal:

During HNSW navigation:

For each candidate node:

- Check if it passes the metadata filter
    - If yes: consider it as a potential result
    - If no: skip it, but still use it for navigation

Result: exact top-K filtered results, fast

This is significantly harder to implement but gives the best results. It's one reason Qdrant is popular for production use cases where filtering is essential.

How to Choose

Situation → Choose

─────────────────────────────────────────────────

Prototype / hackathon → Chroma

Already using PostgreSQL → pgvector

Production, self-hosting → Qdrant

No ops team, need managed → Pinecone

Complex data model, GraphQL fan → Weaviate

Cost-sensitive, small scale → pgvector or Chroma

Billion-scale, serious budget → Qdrant Cloud or Pinecone

My recommendation for most people starting with RAG: Start with Chroma (easy), graduate to Qdrant (production-ready, open source, self-hostable).

🔬

Practical Takeaways

A vector DB is more than search — it's storage, filtering, replication, and APIs around HNSW

Metadata filtering is the hard part — ask how your DB handles filtered search before choosing

Chroma for prototyping, Qdrant for production — you'll outgrow Chroma, but it's the fastest start

pgvector if you're already on Postgres — adding vectors to your existing DB avoids new infrastructure

Pinecone for zero-ops — when you don't want to manage anything

Quantization saves memory — Qdrant's scalar quantization cuts memory by 4x with minimal recall loss

🛡️

What's Next?

Episode 19: Chunking Strategies — Before vectors go into the database, your documents need to be split into chunks. Fixed-size? Semantic boundaries? Recursive? Overlapping? The chunk strategy you pick determines your entire RAG quality.

← Previous

Ep 17: HNSW

Next →

Ep 19: Chunking Strategies

Next: Episode 19 — Chunking Strategies

You've got a 50-page PDF. You can't embed it as one vector. How you split it changes everything about retrieval quality.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.