Context Window: Why 128K Tokens Matters (And Why Conversations Get Dumber)

You're 30 minutes into a conversation with ChatGPT. It's brilliant. Insightful. Remembering everything you said.

45 minutes in — it forgets your name. Repeats something you already discussed. Gives advice that contradicts what it said 10 messages ago.

You didn't get dumber. The AI didn't "get tired." It ran out of context window.

This is the single most misunderstood concept in AI — and the one that affects every conversation you have.

What Is the Context Window?

The context window is the AI's short-term memory. It's the total number of tokens the model can "see" at once — your messages, its replies, system instructions, everything.

Think of it like a whiteboard. The model can only work with what's written on the whiteboard. Once it's full, old stuff gets erased to make room for new stuff.

Model Context Window Approximate Words

GPT-3.5 4,096 tokens ~3,000 words

GPT-4 8,192 tokens ~6,000 words

GPT-4 Turbo 128,000 tokens ~96,000 words

128K tokens ≈ a 300-page book. 1M tokens ≈ an entire Harry Potter series. Sounds massive, right?

It fills up faster than you think.

How the Context Window Fills Up

Here's what most people miss: both sides count. Your messages AND the AI's responses eat into the same window.

A typical conversation:

System prompt: ~500 tokens

Your first message: ~100 tokens

AI's response: ~400 tokens

Your second message: ~150 tokens

AI's response: ~600 tokens

...

After 20 back-and-forth exchanges, you might be at 15,000-20,000 tokens. If the AI writes long responses (and it loves to), you burn through context fast.

Here's a real breakdown for a coding session:

Content Tokens

Twenty thousand tokens — from just 4 exchanges. With a 128K window, you have 107K left. Comfortable. But with GPT-3.5's 4K window? You're already toast.

🔧

Why Conversations Get Dumber

This is the part nobody explains well.

When the context window fills up, the model doesn't crash. It doesn't throw an error. It silently drops old messages. The API trims from the beginning of the conversation — your earliest messages disappear.

But from YOUR perspective, you remember everything. You said "I'm building a React app for a hospital" in message 1. By message 40, that's been evicted from context. Now the AI doesn't know it's a React app. Doesn't know it's for a hospital. It starts giving generic advice. You think it got "dumber" — actually, it just lost its memory.

The Degradation Pattern

Messages 1-10: Full context. AI is sharp. Remembers everything.

Messages 10-30: Context filling up. Still good, but subtle misses.

Messages 30-50: Old context evicted. AI "forgets" early decisions.

Messages 50+: Only recent messages in context. Feels like talking

to someone with amnesia.

This is why the #1 productivity tip for AI is: start a new conversation when the topic shifts. Fresh context = sharp AI.

📊

KV Cache: The Hidden Performance Killer

Every time the model generates a token, it needs to "attend" to every previous token in the context. This isn't free — it requires storing intermediate calculations called the KV cache (Key-Value cache).

From Episode 4, you know the model re-reads all previous tokens for each new prediction. The KV cache stores the results of that reading so it doesn't have to redo it from scratch.

How Big Is the KV Cache?

For a 70B parameter model with 128K context:

KV cache size ≈ 2 × num_layers × num_kv_heads × head_dim × context_length × bytes_per_value

For Llama 3.1 70B (uses GQA with 8 KV heads):

= 2 × 80 × 8 × 128 × 128,000 × 2 bytes (FP16)

≈ 42 GB

40 GB of VRAM — just for the KV cache. That's on top of the model weights themselves (which take ~140 GB in FP16). This is why running long-context models requires enormous GPUs.

The KV cache grows linearly with context length. Double the context, double the cache. This is why a model that's fast at 4K tokens becomes painfully slow at 128K.

Why More Context ≠ Better Answers

Here's the counterintuitive part: bigger context windows can actually hurt quality.

Research from multiple labs (including the famous "Lost in the Middle" paper by Liu et al., 2023) shows that LLMs are best at using information at the beginning and end of the context. Stuff in the middle? Often ignored.

Context position: [Start] .... [Middle] .... [End]

Attention quality: HIGH LOW HIGH

This is called the "Lost in the Middle" problem. If you paste a 50-page document and ask a question about something on page 25, the model is more likely to miss it than something on page 1 or page 50.

Practical implications:

Put the most important information first — don't bury it after 10 pages of context

Repeat key constraints at the end — "Remember, this must be in TypeScript, not JavaScript"

Chunk large documents — process 10 pages at a time instead of all 200 at once

Use retrieval (RAG) instead of stuffing — search for the relevant 500 tokens, don't pass all 100K

Context Window Strategies for Real Work

Strategy 1: Conversation Pruning

Instead of letting the API auto-trim, be intentional:

python

Bad: Just keep appending messages forever

messages.append({"role": "user", "content": new_message})

Good: Summarize old messages when context gets long

if count_tokens(messages) > MAX_CONTEXT * 0.75:

summary = summarize(messages[:-10])  # Keep last 10 messages
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "assistant", "content": f"Summary of our conversation: {summary}"},
        *messages[-10:]  # Recent messages in full
    ]

Strategy 2: System Prompt Economy

Your system prompt is in EVERY request. A 2,000-token system prompt in a 4K context model leaves you only 2K tokens for actual conversation. Keep system prompts tight:

❌ Bad (500 tokens):

"You are a helpful AI assistant created by [company]. You should always

be polite, helpful, and thorough. Please format your responses using

markdown. When writing code, always include comments..."

"Senior Python dev. Concise. Code with comments. Markdown format."

Strategy 3: Context Window as Architecture Decision

Use Case Context Need Model Choice

Don't pay for 128K context if your use case only needs 8K. The KV cache cost is real.

🔬

Try It Yourself — Token Counter

Want to see how fast your context fills up? Here's a quick Python script:

python

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4")

Count tokens in a string

text = "Your message here"
tokens = enc.encode(text)
print(f"Tokens: {len(tokens)}")

Estimate conversation context usage

conversation = [
{"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing in simple terms"},
    {"role": "assistant", "content": "Quantum computing uses quantum bits..."},

]

total = sum(len(enc.encode(msg["content"])) for msg in conversation)
print(f"Total context used: {total} tokens")
print(f"Remaining (128K): {128000 - total} tokens")

🛡️

Practical Takeaways

The context window is the model's total working memory — your messages, its replies, system prompts all compete for the same space

Conversations degrade because old messages get silently dropped — start fresh when topics shift

KV cache is the real bottleneck — it grows linearly with context length and eats VRAM fast

"Lost in the Middle" is real — put critical info at the start or end, not buried in the middle

Don't waste context on bloated system prompts — every token counts

Match context window to your use case — 128K is overkill for a chatbot, essential for document analysis

📦

What's Next?

Episode 6: Temperature, Top-p, Top-k — Why does the same prompt give different answers every time? What makes AI "creative" vs "predictable"? These three knobs control everything — and most people set them wrong.

← Previous

Ep 4: Training vs Inference

Next →

Ep 6: Temperature, Top-p, Top-k

Next: Episode 6 — Temperature, Top-p, Top-k

Ask ChatGPT 'Write me a poem about rain' ten times. You'll get ten different poems. Here's why — and how to control it.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.