Prompt Caching: How to Cut Your AI Costs by 90%

Every time you call an LLM API, the model re-reads your entire system prompt. Every. Single. Time.

That 2,000-token system prompt you carefully crafted? The model processes it fresh on every request. Those 50 tool definitions? Re-parsed from scratch. That RAG context you injected? Re-embedded into the model's attention.

For an agent making 20 calls per task, that's 20 × 2,000 = 40,000 tokens of repeated system prompt processing. At $2.50/million input tokens, you're paying to re-read the same instructions 20 times.

Prompt caching fixes this. The provider caches the processed form of your prompt prefix, and subsequent requests that share the same prefix skip the processing. Same result, fraction of the cost.

How Prompt Caching Works

The concept is simple:

Request 1:

┌────────────────────────────────┬──────────────┐

│ System prompt + tools (2,000t) │ User msg (50t)│ ← Full processing

└────────────────────────────────┴──────────────┘

Cost: 2,050 tokens × full price

Request 2 (same system prompt):

┌────────────────────────────────┬──────────────┐

│ CACHED (2,000t) │ User msg (60t)│ ← Only process new part

└────────────────────────────────┴──────────────┘

Cost: 2,000 tokens × 10% + 60 tokens × full price

The provider stores the processed KV-cache (key-value cache from the attention mechanism) for the shared prefix. When a new request starts with the same prefix, it loads the cached state instead of recomputing it.

What Gets Cached

The cache matches from the beginning of the messages array. As long as the prefix is identical (byte-for-byte), the cache hits:

python

Request 1

messages = [
{"role": "system", "content": "You are a helpful assistant..."},  # ← Cached
    {"role": "user", "content": "What is Python?"}                     # ← New

]

Request 2 (cache HIT — same system prompt)

messages = [
{"role": "system", "content": "You are a helpful assistant..."},  # ← Cache hit!
    {"role": "user", "content": "What is JavaScript?"}                 # ← New

]

Request 3 (cache MISS — different system prompt)

messages = [
{"role": "system", "content": "You are a code reviewer..."},      # ← Different! Cache miss
    {"role": "user", "content": "Review this code"}

]

Anthropic's Prompt Caching

Anthropic was first to make prompt caching explicit and developer-controlled. You mark which parts of the prompt should be cached using cache_control breakpoints.

How to Use It

python

import anthropic

client = anthropic.Anthropic()
response = client.messages.create(
model="claude-sonnet-4-20250514",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": "You are a senior Python developer assistant. " * 100,  # Long system prompt
            "cache_control": {"type": "ephemeral"}  # ← Cache this!
        }
    ],
    messages=[
        {"role": "user", "content": "Write a hello world function"}
    ]

)

Check cache performance

print(response.usage)

{

"input_tokens": 50,

"cache_creation_input_tokens": 2000, ← First call: cache created

"cache_read_input_tokens": 0

}

Second call with the same system prompt:

python

Same system prompt, different user message

response2 = client.messages.create(
model="claude-sonnet-4-20250514",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": "You are a senior Python developer assistant. " * 100,
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {"role": "user", "content": "Now write a fibonacci function"}
    ]

)

print(response2.usage)

{

"input_tokens": 50,

"cache_creation_input_tokens": 0,

"cache_read_input_tokens": 2000, ← Cache HIT! 90% cheaper

}

Anthropic Cache Rules

Rule Detail

Cache Economics

Scenario Without Cache With Cache Savings

The first call is slightly more expensive (cache write fee), but every subsequent call is 90% cheaper. By the 3rd call, you're already saving money.

🔧

OpenAI's Automatic Caching

OpenAI takes a different approach — automatic caching with no developer action required.

How It Works

python

Just make normal API calls — caching happens automatically

response = client.chat.completions.create(
model="gpt-4o",
    messages=[
        {"role": "system", "content": very_long_system_prompt},  # Auto-cached
        {"role": "user", "content": "Hello!"}
    ]

)

Check if cache was used

print(response.usage)

{

"prompt_tokens": 2050,

"completion_tokens": 20,

"prompt_tokens_details": {

"cached_tokens": 2000 ← Automatic cache hit!

}

}

No cache_control, no explicit breakpoints. OpenAI detects repeated prefixes and caches them automatically.

OpenAI Cache Rules

Rule Detail

Anthropic vs OpenAI Caching

Feature Anthropic OpenAI

Control Explicit (cache_control) Automatic

Anthropic's is more aggressive on savings but requires developer effort. OpenAI's is zero-config but less generous on discounts.

📊

What to Cache (Design Patterns)

Pattern 1: Cache System Prompt + Tools

The most common pattern. Your system prompt and tool definitions are the same across all requests:

python

This prefix is identical across ALL user requests → perfect for caching

SYSTEM_PROMPT = """

You are an AI assistant for our e-commerce platform.

You can help customers with orders, returns, product info, and account issues.

Rules

• Always be polite and professional
• Never share customer PII
• Escalate to human support if the customer is angry

...

[2,000 tokens of instructions]

"""

TOOLS = [

# 20 tool definitions, ~3,000 tokens total
    {"type": "function", "function": {"name": "lookup_order", ...}},
    {"type": "function", "function": {"name": "process_return", ...}},
    ...

]

Total cached prefix: ~5,000 tokens

At 100 requests/day: saves ~$1.50/day with Anthropic

Pattern 2: Cache RAG Context

If multiple users ask about the same document, cache the document content:

python

Cache the document content, vary only the question

messages = [
{
        "role": "system",
        "content": f"Answer questions about this document:\n\n{document_text}",
        "cache_control": {"type": "ephemeral"}  # Anthropic
    },
    {"role": "user", "content": user_question}  # Different per user

]

Pattern 3: Cache Conversation History

In multi-turn conversations, the history grows but the prefix stays the same:

python

Turn 1: [system, user1] → NEW

Turn 2: [system, user1, assistant1, user2] → system+user1 CACHED

Turn 3: [system, user1, assistant1, user2, assistant2, user3] → more CACHED

Each turn re-processes less new content

Pattern 4: Stable Prefix, Variable Suffix

Structure your prompts so the stable parts come first:

python

GOOD — stable prefix, variable suffix

messages = [
{"role": "system", "content": long_stable_instructions},     # Cached ✅
    {"role": "system", "content": tool_definitions},              # Cached ✅
    {"role": "system", "content": f"Today's date: {today}"},     # Changes daily ❌
    {"role": "user", "content": user_message}                     # Changes per request ❌

]

BAD — variable content mixed into prefix

messages = [
{"role": "system", "content": f"Today is {today}. {long_instructions}"},  # Never cached!
    {"role": "user", "content": user_message}

]

Key insight: Put anything that changes (dates, user info, dynamic context) AFTER the stable prefix. Anything before the first change gets cached; anything after does not.

Measuring Cache Performance

Anthropic

python

usage = response.usage
total_input = usage.input_tokens
cached = usage.cache_read_input_tokens
cache_written = usage.cache_creation_input_tokens
hit_rate = cached / (total_input + cached + cache_written) * 100
print(f"Cache hit rate: {hit_rate:.1f}%")

OpenAI

python

usage = response.usage
cached = usage.prompt_tokens_details.cached_tokens
total = usage.prompt_tokens
hit_rate = cached / total * 100
print(f"Cache hit rate: {hit_rate:.1f}%")

Track these over time. A healthy application should see 60-90% cache hit rates. If it's below 30%, your prompts might have too much variable content in the prefix.

Common Mistakes

1. Putting dynamic content at the start

python

BAD — timestamp breaks cache every second

{"role": "system", "content": f"Current time: {datetime.now()}\n{instructions}"}

GOOD — timestamp at the end

{"role": "system", "content": f"{instructions}\nCurrent time: {datetime.now()}"}

1. Randomizing prompt order

python

BAD — shuffling tool definitions changes the prefix

random.shuffle(tools)

GOOD — keep tools in consistent order

tools.sort(key=lambda t: t["function"]["name"])

1. Forgetting cache TTL

If requests are more than 5 minutes apart, the cache expires.

For low-traffic applications, caching won't help much.

1. Too-short prefixes

Anthropic needs 1,024+ tokens to cache.

A 200-token system prompt won't be cached — pad it with detailed instructions.

🔬

Practical Takeaways

Put stable content first — system prompts, tool definitions, reference docs → then dynamic content

Anthropic: explicit control with 90% savings — use cache_control for maximum benefit

OpenAI: automatic with 50% savings — zero effort, just design your prompt well

Cache breaks on first differing byte — keep prefixes perfectly consistent

First call costs slightly more (Anthropic) — but pays back by the 3rd call

Track cache hit rates — aim for 60-90%, investigate if below 30%

🛡️

What's Next?

Episode 39: Rate Limiting & Token Budgets — You've optimized costs with caching. Now you need to survive rate limits. TPM, RPM, 429 errors, retry strategies, and how to build production systems that don't crash when they hit the wall.

← Previous

Ep 37: Streaming

Next →

Ep 39: Rate Limiting & Token Budgets

Next: Episode 39 — Rate Limiting & Token Budgets

Ten users? Flawless. A hundred? Smooth. Two thousand hit simultaneously? HTTP 429. Welcome to rate limiting.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.