This is the most fundamental concept in AI engineering. Without understanding tokens, nothing else makes sense.
What Are Tokens?
Let's look at it directly. OpenAI has a tokenizer tool — whatever text you type in, it shows you exactly how AI breaks it down.
Try I am building an application — depending on the model, "application" might be 1 or 2 tokens. "Artificial Intelligence" — 2 tokens. Easy.
| Token # | Text Piece | Type |
|---|---|---|
| 1 | Hello | Word |
| 2 | , | Punctuation |
| 3 | how | Word |
| 4 | are | Word |
| 5 | you | Word |
| 6 | ? | Punctuation |
Here's Where It Gets Interesting
Type the same sentence in Hindi: मुझे एक ऐप बनाना है
Same meaning. But count the tokens — almost double or triple compared to English. Bengali? Even more. Japanese? Almost every character becomes its own token.
Why non-English costs more
The tokenizer was trained on English text. English words are in its vocabulary. Hindi, Bengali, Japanese — the tokenizer has to break these down character by character. That's why non-English languages use significantly more tokens for the same meaning.
Same request. Hindi uses more tokens = more cost. A single chat? 500–1,000 tokens — cheap. But a coding agent making 100 API calls per session, each with 5,000–10,000 tokens? That's $600–1,500 per month. Understanding tokens = understanding your bill.
Prices shown are GPT-4o input rates as of March 2026. Always check your provider's current pricing — rates change frequently.
How Was the Tokenizer Built? (BPE)
The algorithm behind tokenization is called BPE — Byte Pair Encoding. Sounds complex, but the concept is simple.
Every letter is one token. h-e-l-l-o = 5 tokens.
Which 2 characters appear together most? t + h → "the", "that", "this". Merge th into a single token.
th + e appear together a lot? Merge them. Now the is one token.
ing → one token · tion → one token · Hello → one token
This ran on billions of English texts. That's why common English words = 1 token. But बनाना? Broken down character by character.
GPT-4o vocabulary: ~200,000 tokens — 200,000 common subwords learned from training data. Mostly English. This is why English is inherently cheaper and more efficient to process.
Try It Yourself: Python Token Counter
Here's a simple script to count tokens and estimate API costs:
# Install first: pip install tiktoken
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
text = input("Enter text: ")
tokens = enc.encode(text)
print(f"Text: {text}")
print(f"Tokens: {len(tokens)}")
print(f"Token IDs: {tokens}")
# Cost calculator (GPT-4o prices as of March 2026)
input_cost = (len(tokens) / 1_000_000) * 2.50
print(f"API cost (input): ${input_cost:.6f}")Context Window: The Real Limit
When a company says "our model supports 128K tokens" — it means the model can hold a maximum of 128,000 tokens in one conversation.
Think of the context window as a room. 128K tokens = the size of the room. Everything — instructions, conversation history, AI response — must fit inside.
Input vs. output limits
Newer models (GPT-4o, Claude 3.5+) have separate input and output limits — for example, 128K input but only 4K–8K output. Always check both numbers.
| Language | 1,000 tokens ≈ | 128K context ≈ | Effective value |
|---|---|---|---|
| 🇬🇧 English | ~750 words | ~384 pages (a full book) | Full price |
| 🇮🇳 Hindi | ~250–300 words | ~100–130 pages | 3× more expensive |
| 💻 Code | ~200–400 lines | ~50,000 lines | 2× more expensive |
Nobody tells you this: Same model, same price — but Hindi users are effectively paying 3× more for less context. If you give AI a long Hindi document, the context fills fast. The model "forgets" earlier parts — not because it's dumb, but because the room is full.
This becomes critical in RAG (Retrieval Augmented Generation) — where chunking strategy depends entirely on this token math. Wrong chunk size = wrong results. We'll cover that in Module 4.
4 Practical Tips to Optimize Token Usage
Next: Episode 2 — Transformers. In 2017, a paper called "Attention Is All You Need" changed the world. Next, we'll understand how. This is part of a 98-episode series covering AI engineering from tokens to production deployment.