BPE Tokenizer: How AI Builds Its Vocabulary From Scratch

You type "unhappiness" into ChatGPT. The model doesn't see a word. It sees ["un", "happ", "iness"].

Why those three pieces? Why not split it letter by letter? Why not keep it as one chunk? Because an algorithm called Byte Pair Encoding (BPE) decided — months before you ever typed anything — exactly how to carve up every possible string of text. And that decision affects everything: how fast the model runs, how much it costs you per API call, and how well it understands your language.

BPE is where the token economy begins. Let's crack it open.

What Problem Does BPE Solve?

Here's the dilemma. You need to convert text into numbers for a neural network. You have two extreme choices:

Approach Vocabulary Size Problem

BPE sits in the sweet spot. It builds a vocabulary of subword units — pieces that are bigger than characters but smaller than full words. Typical vocab size: 32,000–100,000 tokens.

The result? Common words like "the" get one token. Rare words like "cryptocurrency" get split into familiar pieces: ["crypt", "ocurrency"]. And even gibberish like "asdfghjkl" can be encoded — just character by character as a fallback.

No unknown tokens. Ever.

The BPE Algorithm: Step by Step

BPE is embarrassingly simple. Here's the entire algorithm:

Step 0: Start With Bytes

Begin with a vocabulary of 256 entries — one for each possible byte value. Every string in existence can already be tokenized at this point (one byte = one token). It's just horribly inefficient.

Step 1: Count All Pairs

Take your training corpus (say, a few hundred GB of text). Split everything into bytes. Now count how often each adjacent pair appears.

Training text: "low low low low lowest lowest newer newer wider"

Byte sequences:

l o w → appears 6 times (4 from "low", 2 from "lowest")

o w → appears 6 times

l o → appears 6 times

e s → appears 2 times

e r → appears 4 times

...

Step 2: Merge the Most Frequent Pair

Find the pair that appears most often. Let's say it's (l, o). Create a new token lo and add it to the vocabulary.

Vocabulary: [all 256 bytes] + ["lo"]

Now re-encode the entire corpus using this new token:

Before: l o w l o w l o w l o w l o w e s t l o w e s t n e w e r ...

After: lo w lo w lo w lo w lo w e s t lo w e s t n e w e r ...

Step 3: Repeat

Count pairs again. Now (lo, w) is the most common. Merge it:

Vocabulary: [256 bytes] + ["lo", "low"]

Corpus: low low low low low e s t low e s t n e w e r ...

Next iteration: maybe (e, r) → er. Then (n, ew) → new. Then (low, est) → lowest.

Step 4: Stop When You Hit the Target Size

Keep merging until your vocabulary reaches the desired size. GPT-4 uses 100,256 tokens. Llama 3 uses 128,000 tokens.

Each merge is recorded as a merge rule:

python

Example merge rules (in order)

merge_rules = [
("l", "o") → "lo",      # Rule 1
    ("lo", "w") → "low",    # Rule 2
    ("e", "r") → "er",      # Rule 3
    ("n", "ew") → "new",    # Rule 4
    ("low", "est") → "lowest",  # Rule 5
    # ... 100,000+ rules

]

That's the entire algorithm. Count pairs, merge the top one, repeat.

🔧

How Tokenization Works at Runtime

Once you have the vocabulary and merge rules, tokenizing new text is straightforward:

Split the input into bytes

Apply merge rules in order (rule 1 first, then rule 2, etc.)

Return the resulting tokens

python

Tokenizing "lowest" with our rules above:

Step 0: ['l', 'o', 'w', 'e', 's', 't']

Rule 1 (l+o→lo): ['lo', 'w', 'e', 's', 't']

Rule 2 (lo+w→low): ['low', 'e', 's', 't']

Rule 3 (e+r→er): no match, skip

Rule 4 (n+ew→new): no match, skip

Rule 5 (low+est→lowest): need 'est' first... skip for now

...later rule (e+s→es): ['low', 'es', 't']

...later rule (es+t→est): ['low', 'est']

...Rule 5 (low+est→lowest): ['lowest']

Final tokens: ['lowest'] → token ID [4521]

The order of merge rules matters. It's a priority system — earlier merges were more common in the training data.

📊

Real BPE in Action: GPT-4's Tokenizer

Let's see how OpenAI's tiktoken actually tokenizes text:

python

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4")

Common English

tokens = enc.encode("Hello, how are you?")
print(tokens)  # [9906, 11, 1268, 527, 499, 30]
print(len(tokens))  # 6 tokens

Hindi

tokens = enc.encode("नमस्ते, आप कैसे हैं?")
print(len(tokens))  # 18 tokens!

Code

tokens = enc.encode("def fibonacci(n):\n    return n if n <= 1 else fibonacci(n-1) + fibonacci(n-2)")
print(len(tokens))  # 28 tokens

Gibberish still works

tokens = enc.encode("xyzzyplugh")
print(len(tokens))  # 4 tokens — broken into known subwords

Notice: Hindi uses 3x more tokens than English for roughly the same meaning. That's because GPT-4's training data was mostly English, so English subwords got merged more aggressively. This has real cost implications — Hindi users pay more per API call for the same conversation.

The Vocabulary Matters More Than You Think

Your choice of BPE vocabulary affects everything downstream:

Factor Small Vocab (32K) Large Vocab (128K)

This is why Llama 3 jumped from 32K vocab (Llama 2) to 128K vocab — they wanted better multilingual and code performance.

The Pre-tokenization Step

Before BPE runs, most implementations split text at natural boundaries to prevent weird cross-word merges:

python

GPT-4 pre-tokenization regex (simplified)

import regex

pattern = r"""'s't're've'm'll'd?\w+?\d+?[^\s\w\d]+\s+"""

This ensures:

"I'm unhappy" → ["I", "'m", " un", "happy"]

NOT: ["I'", "m ", "unhap", "py"]

Pre-tokenization prevents the BPE algorithm from merging across word boundaries (like merging the space at the end of one word with the start of the next). Without it, you'd get tokens like "e t" spanning two words — terrible for the model to learn from.

Try It Yourself

python

Minimal BPE implementation in Python

from collections import Counter

def get_pairs(tokens):

"""Count adjacent pairs in token list."""
    pairs = Counter()
    for i in range(len(tokens) - 1):
        pairs[(tokens[i], tokens[i+1])] += 1
    return pairs

def merge_pair(tokens, pair, new_token):

"""Replace all occurrences of pair with new_token."""
    result = []
    i = 0
    while i < len(tokens):
        if i < len(tokens) - 1 and (tokens[i], tokens[i+1]) == pair:
            result.append(new_token)
            i += 2
        else:
            result.append(tokens[i])
            i += 1
    return result

Training corpus as bytes

corpus = list("low low low lowest newest")
print(f"Initial: {corpus}")
print(f"Vocab size: {len(set(corpus))}")

Run 5 BPE merges

merge_rules = []

for step in range(5):

pairs = get_pairs(corpus)
    if not pairs:
        break
    best_pair = pairs.most_common(1)[0][0]
    new_token = best_pair[0] + best_pair[1]
    corpus = merge_pair(corpus, best_pair, new_token)
    merge_rules.append((best_pair, new_token))
    print(f"Merge {step+1}: {best_pair} → '{new_token}' | Tokens: {len(corpus)}")

print(f"\nFinal vocab: {sorted(set(corpus))}")
print(f"Merge rules: {merge_rules}")

Run it. Watch the vocabulary grow one merge at a time.

🔬

Practical Takeaways

BPE builds vocabulary by repeatedly merging the most frequent adjacent pair — that's the entire algorithm

Vocabulary choice affects cost — languages underrepresented in training data use 2-4x more tokens

The merge rules are ordered — earlier rules handle more common patterns, later rules handle rarer ones

Pre-tokenization prevents cross-word merges — regex splits happen before BPE

Bigger vocab = shorter sequences but larger embedding table — it's a tradeoff, not a free lunch

Once trained, the tokenizer is frozen — you can't change it without retraining the model

🛡️

What's Next?

Episode 72: SentencePiece vs tiktoken — Two completely different philosophies for building tokenizers. One treats text as raw bytes, the other needs pre-tokenization. One dominates open-source models, the other powers OpenAI. We'll compare them head-to-head and see why multilingual models overwhelmingly choose SentencePiece.

← Previous

Ep 70: The AI Chip War

Next →

Ep 72: SentencePiece vs tiktoken

Next: Episode 72 — SentencePiece vs tiktoken

Every AI model has a tokenizer. But not all tokenizers are built the same way — and the differences matter more than you think.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.