Cost Optimization: Stop Burning Money on AI Inference

AI is cheap until it isn't.

A side project using GPT-4 costs ₹500/month. The same project at scale costs ₹5,00,000/month. The code doesn't change. The usage does. And the bill sneaks up on you like a gully cricket bet you forgot you made.

If you're building with AI, cost optimization isn't a nice-to-have. It's survival. Here's how to cut your inference costs by 50-90% without sacrificing quality.

The Cost Stack: Where Your Money Goes

Before optimizing, understand the anatomy of an AI bill:

Cost Driver Typical % Optimization Potential

Strategy 1: Smart Model Selection

Not every task needs GPT-4. Not even close.

The Model Ladder

Simple tasks → Haiku, Gemini Flash, GPT-3.5

Medium tasks → Sonnet, GPT-4-turbo

Complex tasks → GPT-4, Opus

Routing logic:

python

def select_model(prompt: str, complexity_hint: str = "auto") -> str:
if complexity_hint == "simple":
        return "claude-haiku-3-5"
# Auto-detect based on prompt characteristics
    word_count = len(prompt.split())
    has_code = any(kw in prompt.lower() for kw in ["code", "function", "bug"])
if word_count < 100 and not has_code:
        return "gpt-3.5-turbo"  # 10x cheaper
    elif word_count < 500:
        return "claude-sonnet-4"
    else:
        return "gpt-4"  # Fallback for complex tasks

Cost Comparison (per 1M tokens)

Model Input Cost Output Cost Use Case

GPT-4 $30 $60 Complex reasoning

GPT-4-turbo $10 $30 General purpose

Rule of thumb: Start with the cheapest model that can possibly work. Escalate only when it fails.

🔧

Strategy 2: Prompt Compression

Every token costs money. Shorter prompts = cheaper inference.

Techniques

1. Remove fluff

python

BAD - 147 tokens

prompt = """

Hello, I hope you're doing well. I was wondering if you could please help me

with something. I have this document here and I need to understand what it's

about. Could you read it and tell me the main points? Thank you so much!

[document text]

"""

GOOD - 23 tokens

prompt = """

Summarize this document in 3 bullet points:

[document text]

"""

1. Use delimiters efficiently

python

Standard format (compact)

prompt = """TASK: summarize

STYLE: bullet points

MAX_LENGTH: 50 words

CONTENT:

{text}"""

1. Few-shot examples (only when needed)

python

Only include examples if zero-shot fails

examples = get_examples_if_confidence_low(task_type, input_text)
prompt = base_prompt + (examples if examples else "")

Prompt Compression Tools

Tool What It Does Savings

LLMLingua Removes redundant tokens 20-40%

📊

Strategy 3: Aggressive Caching

If you've seen the question before, don't ask the model again.

Types of Caches

1. Exact-match cache

python

import hashlib

from functools import lru_cache

@lru_cache(maxsize=10000)

def cached_completion(prompt_hash: str):

# Only hits API if prompt_hash not seen
    return call_llm(prompt)

def get_completion(prompt: str):

prompt_hash = hashlib.sha256(prompt.encode()).hexdigest()
    return cached_completion(prompt_hash)

1. Semantic cache (covered in Episode 83)

Store by embedding similarity, not exact match. "What's the capital of France?" and "France capital city?" are the same question.

Cache Hit Rates by Use Case

Use Case Typical Hit Rate Monthly Savings

FAQ bot 60-80% 70-80%

Strategy 4: Batch Processing

One API call with 100 prompts costs nearly the same as one prompt. Batching = free parallelism.

How Batching Works

python

WITHOUT batching - 10 API calls

for prompt in prompts:

response = client.completions.create(model="gpt-4", prompt=prompt)

WITH batching - 1 API call, ~60% cost reduction

responses = client.completions.create(
model="gpt-4",
    prompts=prompts,  # Array of prompts
    max_tokens=150

)

When to Batch

Scenario Batch? Notes

Processing user requests in real-time No Latency matters

Nightly document processing Yes Perfect for batching

Email classification Yes Delay is acceptable

Chat responses No Needs immediate reply

Analytics/reporting Yes Process in bulk

Strategy 5: Token Reduction Techniques

Output Token Limits

Always set max_tokens based on expected response size:

python

DON'T - lets model decide (can waste tokens)

response = client.chat.completions.create(
model="gpt-4",
    messages=messages

)

DO - constrain output

response = client.chat.completions.create(
model="gpt-4",
    messages=messages,
    max_tokens=150  # Force concise answers

)

Structured Output

Request specific formats to prevent rambling:

python

prompt = """

Extract the meeting details. Respond ONLY in this JSON format:

{

"date": "YYYY-MM-DD",

"time": "HH:MM",

"attendees": ["name1", "name2"]

}

Text: {input_text}

"""

🔬

Cost Monitoring: Know Where You Stand

You can't optimize what you don't measure.

Basic Tracking

python

import time

class LLMTracker:

def __init__(self):
        self.usage = defaultdict(lambda: {"calls": 0, "tokens": 0, "cost": 0})

def log_call(self, model: str, input_tokens: int, output_tokens: int):
        cost = self.calculate_cost(model, input_tokens, output_tokens)
        self.usage[model]["calls"] += 1
        self.usage[model]["tokens"] += input_tokens + output_tokens
        self.usage[model]["cost"] += cost

def calculate_cost(self, model, input_tok, output_tok):
        rates = {
            "gpt-4": (30/1e6, 60/1e6),  # input, output per token
            "claude-haiku": (0.25/1e6, 1.25/1e6)
        }
        input_rate, output_rate = rates.get(model, (0, 0))
        return (input_tok * input_rate) + (output_tok * output_rate)

Alerts to Set

Daily spend exceeds ₹1000

Single request exceeds ₹50

Token usage spikes 2x from baseline

Cache hit rate drops below 50%

🛡️

Practical Takeaways

Start with the cheapest viable model — Haiku and GPT-3.5 handle 70% of tasks

Compress prompts ruthlessly — Every token is money

Cache everything — 60%+ hit rates are achievable in most applications

Batch non-real-time work — One API call, 100x throughput

Set max_tokens conservatively — Prevent runaway responses

Monitor costs per request type — Know which features are expensive

Implement model routing — Escalate to expensive models only when needed

📦

What's Next?

Episode 56: Monitoring & Observability — Logging, tracing, eval in production, LangSmith, Helicone, and building custom dashboards. Because optimizing costs doesn't matter if you can't see what's happening.

← Previous

Ep 54: Data Privacy in AI Systems

Next →

Ep 56: Monitoring & Observability

Next: Episode 56 — Monitoring & Observability

Your AI app works on your laptop. The demos were great. Then it hits production and you can't see what's happening inside.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.