SentencePiece vs tiktoken: The Tokenizer War You Didn't Know About

Every AI model has a tokenizer. But not all tokenizers are built the same way.

OpenAI uses tiktoken. Google, Meta, Mistral — they use SentencePiece. Two completely different philosophies for chopping text into tokens. One requires pre-processing. The other treats raw bytes as sacred. One dominates closed-source. The other owns open-source.

If you're deploying models, switching between providers, or building multilingual products — this distinction matters more than you think.

The Core Philosophical Difference

Here's the fundamental split:

The biggest difference: SentencePiece doesn't need pre-tokenization. You throw raw text at it — any language, any script — and it figures out the splits. tiktoken requires a regex to split text into chunks first, then applies BPE to each chunk.

Why does this matter? Because pre-tokenization rules are language-specific. English word boundaries don't work for Chinese (no spaces), Japanese (mixed scripts), or Thai (no word delimiters). SentencePiece sidesteps this entirely.

How SentencePiece Handles Text

SentencePiece treats the input as a raw byte stream. No assumptions about words, spaces, or language.

The Space Problem

In normal text, spaces are separators — they're not part of words. But SentencePiece needs to be reversible (tokens → original text, losslessly). So it uses a special character ▁ (Unicode 0x2581) to represent spaces:

Input: "Hello world"

SentencePiece: ["▁Hello", "▁world"]

Input: "New York"

SentencePiece: ["▁New", "▁York"]

Input: "I'm fine"

SentencePiece: ["▁I", "'", "m", "▁fine"]

The ▁ is prepended to tokens that follow a space. This means you can always reconstruct the original text by joining tokens and replacing ▁ with space.

tiktoken's Approach

tiktoken uses regex pre-tokenization first:

python

GPT-4's pre-tokenization pattern (simplified)

pattern = r"""'(?i:[sdmt]llvere)[^\r\n\p{L}\p{N}]?+\p{L}+\p{N}{1,3}?[^\s\p{L}\p{N}]++[\r\n]*\s*[\r\n]\s+(?!\S)\s+"""

This regex splits text into chunks before BPE ever sees it. The spaces are absorbed into the token boundaries by the regex, not by the BPE algorithm itself.

🔧

BPE vs Unigram: The Algorithm Choice

SentencePiece offers two tokenization algorithms. This is a major difference from tiktoken (BPE only).

BPE (Bottom-Up)

We covered this in Episode 71. Start with characters, merge the most frequent pairs repeatedly. Build vocabulary bottom-up.

Unigram (Top-Down)

Unigram works the opposite way:

Start with a huge vocabulary (say, 1 million subwords)

Calculate the probability of each subword appearing in the training corpus

Remove the subword whose removal causes the least increase in total loss

Repeat until vocabulary reaches target size

Step 0: vocab = {a, b, c, ab, bc, abc, ...} — 1M entries

Step 1: Remove "xyz" (removing it barely changes total corpus encoding cost)

Step 2: Remove "qw" (same — rare, low impact)

...

Step N: vocab = 32,000 entries — done

Key Difference: Tokenization Determinism

Here's the subtle but important part:

BPE Unigram

Unigram's ability to produce multiple valid tokenizations is actually a feature. During training, you can randomly sample different splits of the same sentence. This is called subword regularization, and it makes the model more robust — like showing it the same word broken up in different ways.

python

Unigram subword regularization example

sentence = "international"

Training step 1: ["inter", "national"]

Training step 2: ["in", "ter", "nation", "al"]

Training step 3: ["international"]

The model sees all three — becomes robust to different tokenizations

📊

Head-to-Head Comparison

Let's tokenize the same text with both:

python

tiktoken (GPT-4)

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4")
text = "The transformer architecture revolutionized natural language processing."
tokens_tiktoken = enc.encode(text)
decoded = [enc.decode([t]) for t in tokens_tiktoken]
print(f"tiktoken ({len(tokens_tiktoken)} tokens): {decoded}")

['The', ' transformer', ' architecture', ' revolution', 'ized', ' natural', ' language', ' processing', '.']

SentencePiece (Llama 3 tokenizer)

from sentencepiece import SentencePieceProcessor

sp = SentencePieceProcessor(model_file="llama3_tokenizer.model")
tokens_sp = sp.encode_as_pieces(text)
print(f"SentencePiece ({len(tokens_sp)} tokens): {tokens_sp}")

['▁The', '▁transform', 'er', '▁architecture', '▁revolution', 'ized', '▁natural', '▁language', '▁processing', '.']

Similar token counts, but subtle differences in where splits happen. Now try non-English:

python

hindi = "ट्रांसफार्मर आर्किटेक्चर ने प्राकृतिक भाषा प्रसंस्करण में क्रांति ला दी"
print(f"tiktoken: {len(enc.encode(hindi))} tokens")    # ~45 tokens
print(f"SentencePiece: {len(sp.encode(hindi))} tokens") # ~25 tokens

SentencePiece crushes tiktoken on non-English text — because it was designed for multilingual from day one. tiktoken's regex pre-tokenization was optimized for English word boundaries.

Multilingual Performance

This is where the choice really matters:

For the same paragraph of ~50 words in each language.

Why? Three reasons:

No pre-tokenization bias: SentencePiece doesn't assume spaces between words

Byte-fallback: SentencePiece can encode unknown characters as byte sequences instead of

Training data balance: Models using SentencePiece often train on more multilingual data

GPT-4o fixed some of this by expanding tiktoken's vocabulary to 200K tokens and including more multilingual merges. But the architectural advantage remains with SentencePiece for truly polyglot models.

When to Use Which

Choose tiktoken when:

Using OpenAI models (you don't have a choice)

Building English-primary applications

You want blazing fast encoding (Rust implementation)

Token counting for API cost estimation

Choose SentencePiece when:

Training your own model

Building multilingual applications

Using open-source models (Llama, Mistral, Gemma)

You want subword regularization (Unigram mode)

Choose Hugging Face Tokenizers when:

You want a unified API across both approaches

Building production inference pipelines

Need the fastest possible implementation

python

Hugging Face tokenizers — works with both tiktoken and SentencePiece models

from transformers import AutoTokenizer

Loads tiktoken-style

gpt4_tok = AutoTokenizer.from_pretrained("openai-community/gpt2")

Loads SentencePiece

llama_tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3-8B")

Same API for both!

tokens = llama_tok.encode("Hello, world!")

🔬

Try It Yourself

python

Compare tokenizers on your own text

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4")
test_texts = [
"Hello, how are you?",                    # English
    "नमस्ते, आप कैसे हैं?",                    # Hindi
    "こんにちは、お元気ですか？",                    # Japanese
    "def fibonacci(n): return n if n <= 1",    # Code
    "🎉🚀💡🤖",                                  # Emoji
    "supercalifragilisticexpialidocious",       # Long word

]

for text in test_texts:

tokens = enc.encode(text)
    pieces = [enc.decode([t]) for t in tokens]
    print(f"{text[:30]:30s} → {len(tokens):2d} tokens: {pieces}")

🛡️

Practical Takeaways

SentencePiece treats text as raw bytes — no pre-tokenization needed, better for multilingual

tiktoken requires regex pre-processing — fast and efficient, but English-biased by design

Unigram (SentencePiece) enables subword regularization — multiple valid tokenizations as data augmentation

Non-English text costs 2-3x more tokens with tiktoken — directly impacts API bills

Hugging Face Tokenizers wraps both — use it when you need a unified API

The tokenizer is part of the model — you can't swap tokenizers without retraining

📦

What's Next?

Episode 73: Model Formats: GGUF, SafeTensors, ONNX — You've downloaded a model. It comes as a .gguf file. Or .safetensors. Or .onnx. What are these formats? Why do they exist? And which one should you use for inference, fine-tuning, or edge deployment?

← Previous

Ep 71: BPE Tokenizer

Next →

Ep 73: Model Formats

Next: Episode 73 — Model Formats

SafeTensors, GGUF, ONNX, AWQ, EXL2 — five different formats for the same model. Here's what each one does and when to use it.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.