Transformers: The Paper That Changed Everything

You type a sentence. It responds with something that makes sense. But how? How does a machine figure out that when you say "The cat sat on the mat because IT was tired" — the "IT" refers to the cat, not the mat?

The answer is Attention. And it all started with one paper.

The Paper That Started It All

In 2017, eight Google engineers published a paper called "Attention Is All You Need."

15 pages. That's it.

From those 15 pages came:

🤖 ChatGPT (OpenAI)

🔵 Claude (Anthropic)

💎 Gemini (Google)

🦙 Llama (Meta)

The architecture they introduced? The Transformer.

Before transformers, AI processed language one word at a time — sequentially. Read the first word, then the second, then the third. Like reading through a straw. Slow, and by the time you reached the end of a sentence, you'd half-forgotten the beginning.

The big idea

Transformers flipped this completely. They said: look at ALL words at once. In parallel. That single idea changed everything.

The Core Idea: Self-Attention

In Episode 1, we learned that AI doesn't read words — it reads tokens. Today's question is: once the model has those tokens, how does it understand them?

The answer is self-attention — every token looks at every other token and asks: "How relevant are you to me?"

The Classic Example

Consider this sentence:

"The cat sat on the mat because IT was tired."

"IT" refers to what? The cat. Not the mat. You figured that out in one second. But a computer needs a mechanism to make that connection. That mechanism is self-attention.

Here's what happens:

The model looks at the token "IT"

It compares "IT" with every other token in the sentence

It computes an attention score for each pair — how strongly are these two tokens related?

"IT" ↔ "cat" → high score (strongly related)

"IT" ↔ "mat" → low score (weakly related)

"IT" ↔ "tired" → high score (IT was tired)

The model doesn't just do this for "IT" — it does this for every single token, all at once, in parallel. Every word is simultaneously checking its relationship with every other word.

Try It Yourself — Interactive Demo

Here's a live visualization of self-attention. Click any word to see its attention scores:

Self-Attention Visualizer

Click any word to see its attention scores

Click any word to see which other words it pays attention to

Strong

Medium

Weak

How Attention Actually Works (The Math)

Don't worry — this is simpler than it sounds.

Every token in the sentence gets converted into three vectors:

Query (Q) — "What am I looking for?"
Key (K) — "What do I contain?"
Value (V) — "What's my actual information?"

Step 1: Compute Scores

Take the Query of one token and multiply it (dot product) with the Key of every other token.

Score("IT", "cat")   = Query_IT · Key_cat   = high number
Score("IT", "mat")   = Query_IT · Key_mat   = low number
Score("IT", "tired") = Query_IT · Key_tired = high number

The dot product measures similarity. If two vectors point in the same direction → high score → these tokens are related.

Step 2: Scale

Divide all scores by √d (square root of the dimension). This keeps the numbers from getting too large and destabilizing training.

Scaled_Score = Score / √d

// For GPT-4o with dimension 128, that's dividing by ~11.3

Step 3: Softmax

Convert all scores into probabilities that add up to 1.0:

Attention weights for "IT":

→ cat: 0.42 (strongest — IT refers to cat)

→ tired: 0.28 (IT was tired)

→ was: 0.18

→ because: 0.06

→ mat: 0.03 (weak — IT doesn't refer to mat)

→ sat: 0.02

→ on: 0.01

Step 4: Weighted Sum

Multiply each token's Value vector by its attention weight, then sum them all up. This gives "IT" a new representation that contains information about the cat being tired.

New_IT = 0.42 × Value_cat + 0.28 × Value_tired + 0.18 × Value_was + ...

This is the magic

After self-attention, the token "IT" doesn't just represent the word "it" anymore — it carries context from the entire sentence. It knows it refers to the cat. Nobody programs these Q, K, V vectors. The model learns them during training by seeing billions of sentences. It figures out on its own that "IT" and "cat" should have high attention when they're in a coreference relationship.

Multi-Head Attention: Multiple Perspectives at Once

Real transformers don't just run attention once. They run it multiple times in parallel — these are called attention heads.

Why? Because a single attention pass can only capture one type of relationship. But language has many:

Head 1 might learn: syntactic relationships (subject ↔ verb)

Head 2 might learn: coreference ("IT" ↔ "cat")

Head 3 might learn: positional relationships (nearby words)

Head 4 might learn: semantic similarity

GPT-4o: 128 heads per layer

GPT-4o reportedly uses 128 attention heads per layer — 128 different "perspectives" on every token relationship, computed simultaneously. The outputs of all heads are concatenated and projected back into a single vector. This gives the model a rich, multi-faceted understanding of each token's relationship to every other token.

Why Transformers Are Fast: Parallelism

Before transformers, the dominant architectures were RNNs (Recurrent Neural Networks) and LSTMs.

These processed tokens sequentially — one at a time, left to right:

RNN / LSTM (old way)

Token 1 → process → Token 2 → process → Token 3 → process → ...

Like reading a book one letter at a time. By the time you reach word 1,000, you've nearly forgotten word 1.

Transformers process all tokens simultaneously:

Transformer (modern)
Token 1 ⟩
Token 2 ⟩ → process ALL AT ONCE → output
Token 3 ⟩

... ⟩

Why GPUs love transformers

GPUs are designed for parallel computation — thousands of calculations at the same time. RNNs couldn't fully utilize GPUs because each step depended on the previous one. Transformers have no such dependency. Result: Training that would have taken months with RNNs takes weeks with transformers. Inference that would take seconds takes milliseconds.

The Full Transformer Architecture (Simplified)

A transformer layer does two main things:

Self-Attention — every token looks at every other token (what we just learned)

Feed-Forward Network — each token passes through a neural network independently (adds computational depth)

These two steps repeat many times — stacked in layers:

Input tokens

↓

[Self-Attention → Feed-Forward] ← Layer 1
↓
[Self-Attention → Feed-Forward] ← Layer 2
↓
[Self-Attention → Feed-Forward] ← Layer 3
↓
    ... (GPT-4o has ~120 layers)
    ↓

Output predictions

Layers build meaning progressively

Each layer builds a deeper understanding. Early layers might learn basic grammar. Middle layers learn meaning. Later layers learn complex reasoning and nuance.

The Numbers

Model Layers Attention Heads Parameters

GPT-2 48 25 1.5B

GPT-3 96 96 175B

More layers and heads = deeper understanding, but also more compute and cost. This is the tradeoff we'll explore in Episode 3 when we talk about parameters.

Why This Matters for You

Understanding transformers isn't just academic. It directly impacts how you use AI:

1. Long prompts lose context

Self-attention has O(n²) complexity. Double the tokens, quadruple the compute. This is why there's a context window limit.

1. Order matters less than you think

Unlike RNNs, transformers see all tokens at once. Putting important instructions at the start vs. end matters less (though models do have position biases).

1. Prompt engineering is attention engineering

When you write a clear, specific prompt, you're helping the model's attention focus on what matters. Vague prompts = scattered attention = worse outputs.

1. Why AI "hallucinates"

The model generates tokens based on attention patterns learned from training data. If it hasn't seen enough examples of a topic, the attention patterns are weak, and it fills gaps with plausible-sounding but incorrect tokens.

The hallucination root cause

Hallucination isn't a bug — it's a fundamental property of how attention-based generation works. The model doesn't "know" things. It predicts tokens based on patterns. When patterns are weak, it guesses. This is why grounding (RAG, tools, citations) matters so much in production.