Embeddings: Turning Words Into Numbers Machines Actually Understand

Here's a question that should bother you: how does a computer know that "king" is to "queen" as "man" is to "woman"?

Computers don't understand language. They understand numbers. Floating point numbers, specifically. So somewhere between your text and the machine's understanding, there's a translation layer that converts human meaning into math.

That translation layer is called embeddings. And once you understand them, half of modern AI — search, recommendations, RAG, clustering — suddenly makes sense.

The Problem: Computers Can't Read

Traditional programming treated words as strings. "Cat" was just three characters: C-A-T. The computer had no concept that "cat" was closer to "dog" than to "spacecraft."

Early attempts used one-hot encoding:

cat  = [1, 0, 0, 0, 0, ...]  (position 1)
dog  = [0, 1, 0, 0, 0, ...]  (position 2)
moon = [0, 0, 1, 0, 0, ...]  (position 3)

Every word gets its own dimension. With 100,000 words, each vector is 100,000 dimensions long. And the worst part? The distance between "cat" and "dog" is the same as between "cat" and "spacecraft." There's zero concept of meaning.

Embeddings fix this completely.

What an Embedding Actually Is

An embedding is a list of numbers — a vector — that captures the meaning of a piece of text.

python

Conceptual example (real embeddings have 768-3072 dimensions)

"king" = [0.82, -0.41, 0.19, 0.55, -0.73, ...]

"queen" = [0.79, -0.38, 0.21, 0.57, -0.71, ...]

"apple" = [-0.12, 0.91, -0.33, 0.08, 0.44, ...]

Notice something? "King" and "queen" have similar numbers. "Apple" is completely different. The numbers encode meaning.

Each dimension captures some abstract feature — maybe royalty, gender, animacy, concreteness. We don't explicitly design these features. The model learns them during training, just like how a child learns that dogs and cats are both pets without anyone defining the category mathematically.

Dimensions and What They Capture

Real embedding models use 768 to 3072 dimensions. That's a lot of numbers per word. Why so many?

Think of it like this. If you had only 2 dimensions, you could maybe separate animals from objects. But you couldn't distinguish a pet from a wild animal, or a domestic cat from a tiger. More dimensions = more nuance.

Model Dimensions What It Can Capture

🔧

The Word2Vec Revolution

Before 2013, embeddings were hand-crafted features. Then Word2Vec changed everything with one brilliant idea: words that appear in similar contexts have similar meanings.

"The cat sat on the mat." "The dog sat on the mat."

Cat and dog appear in similar contexts → they should have similar vectors.

Word2Vec trained a simple neural network on billions of words, learning 300-dimensional vectors for each word. And then something magical happened.

The King - Man + Woman = Queen Moment

python

vector("king") - vector("man") + vector("woman") ≈ vector("queen")

Nobody told the model about gender. Nobody defined royalty. The model discovered these relationships by reading text. This was the moment the field realized embeddings capture structure in language, not just similarity.

Other examples that worked:

Paris - France + Italy = Rome

walked - walking + swimming = swam

bigger - big + small = smaller

The math of meaning. Just from reading text.

📊

From Words to Sentences: Modern Embeddings

Word2Vec had a fatal flaw: one vector per word. The word "bank" got the same vector whether you meant riverbank or financial bank.

Modern embedding models (2020+) fix this by embedding entire passages, not individual words:

python

Same word, different meanings → different embeddings

embed("I deposited money in the bank") → [0.82, -0.15, ...]  # financial
embed("I sat by the river bank")       → [-0.33, 0.71, ...]  # nature

These models use the transformer architecture (remember Episode 2?) to read the full context before generating the embedding. The entire sentence gets compressed into a single vector.

How the Embedding Is Generated

Tokenize the input text

Pass through transformer layers — each token attends to all others

Pool the output — usually mean-pool all token representations into one vector

Normalize — scale to unit length for cosine similarity

The result: a single vector that captures the meaning of the entire input.

High-Dimensional Space: Where Meaning Lives

Here's where it gets mind-bending. Each embedding is a point in high-dimensional space. With 768 dimensions, every piece of text is a point in 768-dimensional space.

You can't visualize 768 dimensions. But you can understand the principle in 2D:

^ dimension 2
     |
     |   • "I love cricket"
     |   • "Cricket is amazing"
     |
     |              • "The stock market crashed"
     |              • "Financial crisis in Asia"
     |
     +------------------------------------> dimension 1

Similar texts cluster together. Different topics are far apart. This is the entire foundation of semantic search — instead of matching keywords, you find texts that are close in this meaning-space.

Why This Matters for AI Infrastructure

Embeddings are the bridge between human language and machine computation:

Use Case How Embeddings Help

Every single one of these works the same way: embed the text, then do math on the vectors.

Try It Yourself

Generate embeddings using Ollama (free, local):

bash

Install nomic-embed-text

ollama pull nomic-embed-text

Get an embedding

curl http://localhost:11434/api/embeddings -d '{

"model": "nomic-embed-text",

"prompt": "The cat sat on the mat"

}'

Returns: {"embedding": [0.0234, -0.0891, 0.0412, ...]} (768 numbers)

Or using OpenAI's API:

python

from openai import OpenAI

client = OpenAI()
response = client.embeddings.create(
model="text-embedding-3-small",
    input="The cat sat on the mat"

)

vector = response.data[0].embedding
print(f"Dimensions: {len(vector)}")  # 1536
print(f"First 5: {vector[:5]}")      # [0.0234, -0.0891, ...]

Compare two sentences:

python

import numpy as np

def cosine_similarity(a, b):

return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

v1 = get_embedding("I love playing cricket")
v2 = get_embedding("Cricket is my favorite sport")
v3 = get_embedding("The stock market crashed today")
print(cosine_similarity(v1, v2))  # ~0.92 (very similar!)
print(cosine_similarity(v1, v3))  # ~0.15 (totally different)

🔬

Practical Takeaways

Embeddings convert text into vectors — lists of numbers that capture semantic meaning

Similar meanings → similar vectors — this is the foundation of semantic search and RAG

Modern embeddings are contextual — the same word gets different vectors in different contexts

Dimensions = nuance — more dimensions capture finer distinctions (768 to 3072 is typical)

Embeddings are the universal interface — search, RAG, clustering, recommendations all use them

You can run them locally — Ollama + nomic-embed-text gives you free, private embeddings

🛡️

What's Next?

Episode 16: Vector Similarity — You've got these embedding vectors. Now how do you measure which ones are "close"? Cosine similarity, dot product, Euclidean distance — when to use which, and the visual intuition that makes it click.

← Previous

Ep 14: Speculative Decoding

Next →

Ep 16: Vector Similarity

Next: Episode 16 — Vector Similarity

You've turned text into numbers. Every sentence is now a point in 768-dimensional space. Now how do you find what's similar?

This is part of a 98-episode series covering AI engineering from tokens to production deployment.