KV Cache: Why Your AI Conversations Slow Down

Open ChatGPT. Type a message. Get a response in 2 seconds.

Now paste a 20-page document and ask a question about it. Watch the response crawl in at half the speed.

Continue the conversation for another 50 messages. Notice how each response takes longer than the last.

You're watching the KV cache grow. And it's eating your GPU's memory alive.

What Is the KV Cache?

From Episode 2, you know transformers use attention — every token looks at every previous token to decide what's relevant. From Episode 4, you know inference generates one token at a time.

Here's the problem: when generating token #100, the model needs to attend to all 99 previous tokens. That means running the attention computation for ALL previous tokens again. Then for token #101, it repeats for all 100 previous tokens. And so on.

Without optimization, generating a 500-token response from a 500-token prompt would require:

Token 1: Attend to 500 tokens (prompt)

Token 2: Attend to 501 tokens

Token 3: Attend to 502 tokens

...

Token 500: Attend to 999 tokens

Total attention computations: 500 + 501 + 502 + ... + 999 = 374,750

That's wildly redundant. Token 1's attention to the prompt is the same whether you're computing it for token 2 or token 500. Why redo it?

The KV cache stores these intermediate results. Specifically, it caches the Key and Value vectors from the attention mechanism so they don't need to be recomputed.

Keys and Values: The Attention Mechanism

Quick attention refresher (simplified):

For each token, the attention mechanism computes three vectors:

Query (Q): "What am I looking for?"
Key (K): "What do I contain?"
Value (V): "What information do I provide?"
Attention score = how well a Query matches a Key. High match → that token's Value gets more weight in the output.

Token "Paris":

Query: "I need a noun that's a capital city"

Key: "I am a proper noun, a city name"

Value: "Geographic/cultural information about Paris"

Token "France":

Key: "I am a country name"

Value: "European nation, romance language"

Attention("Paris" query × "France" key) = HIGH match

→ France's Value strongly influences Paris's output

What Gets Cached

During generation, only the Keys and Values for previously processed tokens need caching. The Query is only needed for the current token being generated.

Step 1: Process prompt (tokens 1-500)

→ Compute K,V for ALL 500 tokens

→ Store in KV cache

Step 2: Generate token 501

→ Compute Q for token 501 only

→ Compute K,V for token 501, ADD to cache

→ Attend: Q₅₀₁ × [K₁, K₂, ..., K₅₀₁] → attention weights

→ Output: weighted sum of [V₁, V₂, ..., V₅₀₁]

Step 3: Generate token 502

→ Compute Q for token 502 only

→ Compute K,V for token 502, ADD to cache

→ Attend: Q₅₀₂ × [K₁, K₂, ..., K₅₀₂]

→ NO recomputation of K₁...K₅₀₁ — they're cached!

Without the KV cache, every single generated token would require recomputing K and V for ALL previous tokens. With it, you only compute K and V for the NEW token and look up the rest.

🔧

How Much Memory Does It Use?

The KV cache size for one request:

KV cache (bytes) = 2 × num_layers × num_kv_heads × head_dim × seq_len × bytes_per_value

Let's calculate for real models:

For Llama 3.1 70B at full 128K context:

328 MB per 1K tokens × 128 = 41,984 MB ≈ 42 GB per request
42 GB of VRAM for ONE user's conversation at max context. That's more than the model weights themselves in Q4 quantization. Serve 10 users simultaneously? That's 420 GB just for KV caches — before you even load the model weights.

This is why KV cache management is the #1 concern in production inference. It's not compute that kills you — it's memory.

📊

Why Conversations Slow Down

Two compounding effects:

1. Growing Attention Cost

Each new token must attend to ALL cached tokens. As the cache grows, each token generation takes more time:

Context 1K: Attention over 1,000 K vectors → fast

Context 10K: Attention over 10,000 K vectors → slower

Context 100K: Attention over 100,000 K vectors → much slower

The attention computation scales as O(n) with sequence length for each new token. At 128K context, each token generation requires attending to 128,000 previous positions.

1. Memory Bandwidth Saturation

The KV cache lives in VRAM. For each generated token, the entire KV cache must be read:

Llama 70B at 32K context:

KV cache: ~1.3 GB

Read per token: 1.3 GB

A100 bandwidth: 2 TB/s

Time just for KV reads: ~0.65 ms per token

Llama 70B at 128K context:

KV cache: ~42 GB

Read per token: 42 GB

Time just for KV reads: ~2.5 ms per token

At 128K context, reading the KV cache alone takes 4x longer. Add the model weight reads, and generation speed drops significantly.

Cache Eviction Strategies

When the context window is full, something has to go. Different strategies trade off differently:

1. FIFO (First In, First Out)

The simplest: drop the oldest tokens.

Window: [A B C D E F G H] (full at 8 tokens)

New token I arrives:

Result: [B C D E F G H I] (A dropped)

Problem: The system prompt is the oldest — and often the most important. FIFO drops it first.

1. Keep First + Recent

Keep the first N tokens (system prompt) and the most recent M tokens. Drop everything in the middle.

Window: [SYS SYS... dropped ...F G H I]

↑ Protected    ↑ Evicted    ↑ Recent

This is what most production systems use. It preserves the system prompt and recent context while sacrificing middle history.

1. Attention-Based Eviction

Drop the tokens that received the LEAST attention in recent generations. If the model isn't looking at a token, it probably doesn't need it.

Attention scores for each cached token:

Token A: 0.001 (barely attended to)

Token B: 0.150 (important)

Token C: 0.002 (barely attended to)

Token D: 0.200 (very important)

Evict: A, C (lowest attention scores)

Keep: B, D (high attention)

More sophisticated but computationally expensive. Research area, not widely deployed in production yet.

Sliding Window Attention

Mistral popularized a different approach: don't cache everything. Use a fixed-size window of recent tokens.

Full attention (Llama):

Token 100 attends to: [1, 2, 3, ..., 99, 100] (ALL previous)

KV cache: grows linearly

Sliding window (Mistral, window=4096):

Token 100 attends to: [96, 97, 98, 99, 100] (only last 4096)

KV cache: FIXED at 4096 tokens

The KV cache never exceeds the window size. Memory stays constant regardless of conversation length.

"But won't it forget everything outside the window?"

Not entirely. Through the layers of the transformer, information propagates. Token 1's information influenced token 4096, which influenced token 8192. It's like a game of telephone — diluted but not completely lost.

Layer 1: Token N sees tokens [N-4096, ..., N]
Layer 2: Token N indirectly sees tokens [N-8192, ..., N] (through layer 1's outputs)
Layer 32: Token N has indirect access to tokens very far back

Effective context ≈ window_size × num_layers = 4096 × 32 = 131,072

This is why Mistral claims effective context beyond its sliding window size.

🔬

Multi-Query and Grouped-Query Attention

The biggest innovation in reducing KV cache size: share Key/Value heads across multiple Query heads.

Standard Multi-Head Attention (MHA)

Original transformer: every attention head has its own Q, K, and V.

Heads: 64 Query heads, 64 Key heads, 64 Value heads

KV cache: 64 × head_dim × seq_len × 2 = BIG

Multi-Query Attention (MQA)

One shared K/V head for ALL query heads:

Heads: 64 Query heads, 1 Key head, 1 Value head

KV cache: 1 × head_dim × seq_len × 2 = 64x SMALLER

Massive memory savings. Used by PaLM, Falcon, and others. Downside: slight quality loss because all queries attend to the same keys/values.

Grouped-Query Attention (GQA)

The compromise. Group query heads and share K/V within groups:

Heads: 64 Query heads, 8 Key heads, 8 Value heads

(8 groups of 8 queries sharing 1 K/V pair)

KV cache: 8 × head_dim × seq_len × 2 = 8x SMALLER than MHA

Method Query Heads KV Heads KV Cache Reduction Quality

MHA 64 64 1x (baseline) Best

GQA-8 64 8 8x Near-MHA

MQA 64 1 64x Slight loss

Llama 3.1 uses GQA-8 — 64 query heads, 8 KV heads. This is why its KV cache is manageable at 128K context. Without GQA, the cache would be 8x larger.

🛡️

KV Cache Quantization

Just like model weights (Episode 9), the KV cache can be quantized:

FP16 KV cache at 128K context (70B model): ~42 GB

INT8 KV cache at 128K context (70B model): ~21 GB

INT4 KV cache at 128K context (70B model): ~10.5 GB

vLLM supports KV cache quantization:

bash

vllm serve meta-llama/Llama-3.1-70B-Instruct \

--kv-cache-dtype fp8 # Half the KV cache memory

The quality impact is minimal for most tasks — the KV cache stores intermediate computations that are more robust to precision loss than model weights.

📦

Try It Yourself — Measure KV Cache Impact

bash

Using llama.cpp, compare speed at different context lengths

Short context (fast)

echo "Tell me a joke"./llama-cli -m model.gguf -ngl 99 -c 512

Long context (slower)

cat long_document.txt./llama-cli -m model.gguf -ngl 99 -c 32768 \

-p "Summarize this document:"

Watch VRAM usage during generation

watch -n 0.5 nvidia-smi

You'll see VRAM climb as the context grows — that's the KV cache expanding in real time.

🚀

Practical Takeaways

The KV cache stores attention keys and values to avoid redundant computation — without it, inference would be orders of magnitude slower

It grows linearly with context length — 128K context on a 70B model needs ~42 GB per user

KV cache is the main memory bottleneck in production — model weights are fixed, KV cache scales with users × context

Grouped-Query Attention (GQA) reduces cache 8x — that's why modern models (Llama 3, Mistral) use it

Sliding window attention trades memory for information loss — fixed-size cache regardless of conversation length

KV cache quantization (FP8/INT8) is free performance — minimal quality impact, significant memory savings

🔄

What's Next?

Episode 13: Batching & Concurrency — One user is easy. A thousand users at once? That's where continuous batching, dynamic batching, and clever scheduling turn a single GPU into a production server. Throughput vs latency — the fundamental tradeoff of serving AI at scale.

← Previous

Ep 11: Ollama vs vLLM vs TGI

Next →

Ep 13: Batching & Concurrency

Next: Episode 13 — Batching & Concurrency

One user, 500ms response. Beautiful. Now a thousand users hit simultaneously. Welcome to the batching problem.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.