Semantic Caching: Cache by Meaning, Not Exact Match

You've seen exact-match caching. Same question, same answer. Fast, cheap, boring.

But users don't ask the same question twice. They ask:

"What's the capital of France?"

"France capital city?"

"Which city is France's capital?"

Different strings. Same meaning. Same answer.

Semantic caching recognizes this. It caches by meaning, not by characters. And it can cut your AI costs by 40-70%.

The Problem with Exact-Match Caching

Traditional cache:

Key: "What's the capital of France?"

Value: "Paris"

Next query: "What is the capital of France?"

Result: MISS ❌ (different string)

Even tiny variations break the cache:

Extra spaces

Different capitalization

Synonyms

Word order changes

In practice, exact-match caching hits 10-20% of the time for natural language queries. Semantic caching hits 40-70%.

How Semantic Caching Works

The Flow

User Query → Embed → Search Cache (vector similarity) →

If similar > threshold: Return cached response

Else: Call LLM → Store (embedding, response) in cache

Step-by-Step

1. Embed the query

python

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')

def get_embedding(text: str):

return model.encode(text)

query = "What's the capital of France?"
query_embedding = get_embedding(query)  # 384-dimensional vector

1. Search the cache

python

import numpy as np

from sklearn.metrics.pairwise import cosine_similarity

cache = [
{"embedding": [...], "response": "Paris", "query": "France capital?"},
    {"embedding": [...], "response": "Berlin", "query": "Germany capital?"},

]

def find_in_cache(query_emb, threshold=0.92):
for entry in cache:
        similarity = cosine_similarity([query_emb], [entry["embedding"]])[0][0]
        if similarity > threshold:
            return entry["response"], similarity
    return None, 0

1. Return or fetch

python

def get_response(user_query: str):

query_emb = get_embedding(user_query)
    cached_response, similarity = find_in_cache(query_emb)

if cached_response:
        return {"response": cached_response, "source": "cache", "similarity": similarity}

# Cache miss - call the LLM
    llm_response = call_llm(user_query)

# Store in cache
    cache.append({
        "embedding": query_emb,
        "response": llm_response,
        "query": user_query
    })

return {"response": llm_response, "source": "llm"}

🔧

Choosing the Right Threshold

The similarity threshold is critical. Too high: misses good matches. Too low: returns wrong answers.

Threshold Behavior Best For

Domain-Specific Tuning

python

THRESHOLDS = {

"faq": 0.93,        # High precision needed
    "code_help": 0.95,  # Code is precise
    "brainstorming": 0.88,  # More tolerance for variation
    "support": 0.92     # Balance precision and recall

}

📊

Production Implementation with GPTCache

Don't build this from scratch. Use GPTCache:

python

from gptcache import Cache

from gptcache.adapter import openai

from gptcache.embedding import Onnx

from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation

Initialize cache

cache = Cache()

cache.init(

embedding_func=Onnx(),  # Fast, local embedding model
    data_manager=data_manager,
    similarity_evaluation=SearchDistanceEvaluation(),

)

Patch OpenAI to use cache

openai.cache = cache

Now all your OpenAI calls are semantically cached

response = openai.chat.completions.create(
model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "What's the capital of France?"}]

)

Vector Storage Options

Store Best For Scale

Redis Example

python

import redis

import json

import numpy as np

r = redis.Redis(host='localhost', port=6379)

def cache_response(query_emb, response):

# Store as hash: embedding_id -> {embedding, response}
    emb_id = f"emb:{hash(query_emb.tobytes())}"
    r.hset(emb_id, mapping={
        "embedding": query_emb.tobytes(),
        "response": json.dumps(response)
    })

def find_similar(query_emb, threshold=0.92):
# Scan all entries (use vector DB for production scale)
    for key in r.scan_iter(match="emb:*"):
        entry = r.hgetall(key)
        cached_emb = np.frombuffer(entry[b"embedding"])
        similarity = cosine_similarity([query_emb], [cached_emb])[0][0]
        if similarity > threshold:
            return json.loads(entry[b"response"]), similarity
    return None, 0

Advanced Techniques

1. Response Validation

Before returning a cached response, verify it's still relevant:

python

def validate_cache_hit(query: str, cached_response: str) -> bool:
"""Use a cheap model to verify cache hit validity"""
    validation_prompt = f"""
    User asked: {query}
    Cached answer: {cached_response}
Is this cached answer appropriate for the user's question?
    Reply only YES or NO.
    """
    result = call_cheap_llm(validation_prompt)
    return "YES" in result.upper()

1. Cache Warming

Pre-populate cache with common queries:

python

common_queries = [
"How do I reset my password?",
    "What are your business hours?",
    "How do I contact support?"

]

for query in common_queries:

response = call_llm(query)
    embedding = get_embedding(query)
    store_in_cache(embedding, response)

1. Time-Based Expiration

Some responses become stale:

python

def store_with_ttl(query_emb, response, ttl_hours=24):
cache_entry = {
        "embedding": query_emb,
        "response": response,
        "timestamp": time.time(),
        "ttl": ttl_hours * 3600
    }
    # Store in Redis with expiration
    r.setex(f"emb:{id}", ttl_hours * 3600, json.dumps(cache_entry))

🔬

Measuring Cache Performance

Track these metrics:

python

class SemanticCacheMetrics:

def __init__(self):
        self.hits = 0
        self.misses = 0
        self.similarities = []

@property
    def hit_rate(self):
        total = self.hits + self.misses
        return self.hits / total if total > 0 else 0

@property
    def estimated_savings(self):
        # Assuming $0.002 per API call
        return self.hits * 0.002

Expected Results by Use Case

Use Case Hit Rate Monthly Savings

🛡️

When NOT to Use Semantic Caching

❌ Time-sensitive queries — "What's the weather today?"

❌ Personalized responses — "What's my account balance?"

❌ Creative writing — Users want fresh content

❌ Exact data retrieval — SQL queries, specific IDs

📦

Practical Takeaways

Semantic caching hits 3-4x more than exact-match — Same meaning, different words

Threshold tuning is critical — 0.92-0.95 works for most use cases

Use GPTCache — Don't build from scratch

Validate cache hits — Quick LLM check prevents stale responses

Warm the cache — Pre-populate with your top 100 queries

Set TTLs — Some answers expire (pricing, features, policies)

Measure savings — Track hit rates and cost reduction

🚀

What's Next?

Episode 84: AI Gateway Architecture — OpenRouter, LiteLLM, unified APIs, fallbacks, and provider abstraction. The infrastructure layer that makes semantic caching even more powerful.

← Previous

Ep 82: Webhooks & Event-Driven AI

Next →

Ep 84: AI Gateway Architecture

Next: Episode 84 — AI Gateway Architecture

OpenAI, Claude, Gemini, self-hosted Llama — managing all of them is a nightmare. An AI gateway unifies them all.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.