Your app works perfectly in development. Ten users? Flawless. A hundred? Smooth. Then you launch. Two thousand users hit your API simultaneously, and you get:
HTTP 429 Too Many Requests
{
"error": {
"message": "Rate limit reached for gpt-4o. Limit: 30000 TPM. Current: 31247 TPM.",
"type": "rate_limit_error"
}}
Your app is down. Users see errors. Your Slack is on fire.
Rate limits are the invisible wall every AI application hits at scale. Understanding them isn't optional — it's survival.
How Rate Limits Work
AI providers limit your usage along two dimensions:
RPM — Requests Per Minute
How many API calls you can make per minute, regardless of size.
Limit: 500 RPM
Request 1 (10 tokens) ✅ Counted
Request 2 (10,000 tokens) ✅ Counted (same weight as request 1)
Request 3...
...
Request 500 ✅ Last allowed
Request 501 ❌ 429 error
TPM — Tokens Per Minute
How many total tokens (input + output) you can process per minute.
Limit: 150,000 TPM
Request 1: 5,000 input + 2,000 output = 7,000 tokens ✅ Running total: 7,000
Request 2: 10,000 input + 3,000 output = 13,000 tokens ✅ Running total: 20,000
...
Request N: 8,000 tokens ❌ Would exceed 150,000
Actual Limits (2025)
Provider Tier RPM TPM Notes
| OpenAI (GPT-4o) | Tier 1 | 500 | 30,000 | Free tier |
|---|---|---|---|---|
| OpenAI (GPT-4o) | Tier 2 | 5,000 | 450,000 | After $50 spend |
| OpenAI (GPT-4o) | Tier 5 | 10,000 | 30,000,000 | After $1,000+ spend |
| Anthropic (Claude Sonnet) | Tier 1 | 50 | 40,000 | New accounts |
| Anthropic (Claude Sonnet) | Tier 4 | 4,000 | 400,000 | Established accounts |
| Groq (Llama 3) | Free | 30 | 6,000 | Very restrictive |
| Groq (Llama 3) | Paid | 300 | 60,000 | Better but still limited |
The trap: Your development tier might have generous limits, but your production account's limits depend on your spending history. You can be rate-limited at launch even with a paid account.
The 429 Response
When you hit a rate limit, you get a 429 Too Many Requests response:
json
{
"error": {
"message": "Rate limit reached for gpt-4o in organization org-xxx on tokens per min (TPM): Limit 30000, Used 28547, Requested 2891.",
"type": "tokens",
"code": "rate_limit_exceeded"
}}
The response usually includes headers telling you when to retry:
HTTP/1.1 429 Too Many Requests
Retry-After: 20
x-ratelimit-limit-requests: 500
x-ratelimit-remaining-requests: 0
x-ratelimit-reset-requests: 42s
x-ratelimit-limit-tokens: 150000
x-ratelimit-remaining-tokens: 0
x-ratelimit-reset-tokens: 18s
Always read these headers. They tell you exactly how long to wait.
🔧
Retry Strategies
Strategy 1: Exponential Backoff
The most common and most important strategy:
python
import time
import random
from openai import OpenAI, RateLimitError
client = OpenAI()
def call_with_retry(messages, max_retries=5):
for attempt in range(max_retries):
try:
return client.chat.completions.create(
model="gpt-4o",
messages=messages
)
except RateLimitError as e:
if attempt == max_retries - 1:
raise
# Exponential backoff: 1s, 2s, 4s, 8s, 16s
wait = (2 ** attempt) + random.uniform(0, 1)
print(f"Rate limited. Retrying in {wait:.1f}s (attempt {attempt + 1})")
time.sleep(wait)Strategy 2: Respect Retry-After Header
python
import httpx
def call_with_retry_after(messages, max_retries=5):
for attempt in range(max_retries):
try:
return client.chat.completions.create(
model="gpt-4o",
messages=messages
)
except RateLimitError as e:
# Parse Retry-After from headers
retry_after = float(e.response.headers.get("Retry-After", 10))
print(f"Rate limited. Waiting {retry_after}s as instructed by server.")
time.sleep(retry_after)Strategy 3: Token-Aware Rate Limiting
Don't just retry — prevent hitting limits in the first place:
python
import tiktoken
from time import time, sleep
from collections import deque
class TokenRateLimiter:
def __init__(self, tpm_limit=150_000, rpm_limit=500):
self.tpm_limit = tpm_limit
self.rpm_limit = rpm_limit
self.token_log = deque() # (timestamp, token_count)
self.request_log = deque() # (timestamp,)
self.encoder = tiktoken.encoding_for_model("gpt-4o")def estimate_tokens(self, messages):
"""Estimate token count before making the request"""
total = 0
for msg in messages:
total += len(self.encoder.encode(msg.get("content", "")))
total += 4 # Role, formatting overhead
return totaldef wait_if_needed(self, estimated_tokens):
"""Block until we're within rate limits"""
now = time()# Clean old entries (older than 60 seconds)
while self.token_log and self.token_log[0][0] < now - 60:
self.token_log.popleft()
while self.request_log and self.request_log[0] < now - 60:
self.request_log.popleft()# Check TPM
current_tpm = sum(t[1] for t in self.token_log)
if current_tpm + estimated_tokens > self.tpm_limit:
wait_time = 60 - (now - self.token_log[0][0])
print(f"TPM limit approaching. Waiting {wait_time:.1f}s")
sleep(wait_time)# Check RPM
if len(self.request_log) >= self.rpm_limit:
wait_time = 60 - (now - self.request_log[0])
print(f"RPM limit approaching. Waiting {wait_time:.1f}s")
sleep(wait_time)# Log this request
self.token_log.append((time(), estimated_tokens))
self.request_log.append(time())Usage
limiter = TokenRateLimiter(tpm_limit=150_000, rpm_limit=500)def safe_call(messages):
estimated = limiter.estimate_tokens(messages)
limiter.wait_if_needed(estimated)
return client.chat.completions.create(model="gpt-4o", messages=messages)📊
Token Budgets: Controlling Costs
Rate limits protect the provider. Token budgets protect you.
Per-Request Budgets
python
Hard cap on output length
response = client.chat.completions.create(
model="gpt-4o",
messages=messages,
max_tokens=500 # Never generate more than 500 output tokens)
If the model hits this limit, finish_reason = "length"
if response.choices[0].finish_reason == "length":
print("Warning: Response was truncated!")Per-User Budgets
python
class UserBudget:
def __init__(self, daily_limit_usd=1.0):
self.daily_limit = daily_limit_usd
self.usage = {} # user_id → dollars spent todaydef check(self, user_id, estimated_cost):
today = date.today().isoformat()
key = f"{user_id}:{today}"
spent = self.usage.get(key, 0)if spent + estimated_cost > self.daily_limit:
raise BudgetExceeded(
f"Daily limit reached. Spent ${spent:.2f} of ${self.daily_limit:.2f}"
)self.usage[key] = spent + estimated_costdef estimate_cost(self, input_tokens, max_output_tokens, model="gpt-4o"):
rates = {
"gpt-4o": {"input": 2.50, "output": 10.00},
"gpt-4o-mini": {"input": 0.15, "output": 0.60},
"claude-sonnet": {"input": 3.00, "output": 15.00},
}
r = rates[model]
return (input_tokens * r["input"] + max_output_tokens * r["output"]) / 1_000_000budget = UserBudget(daily_limit_usd=5.0)Per-Task Budgets (For Agents)
python
class TaskBudget:
def __init__(self, max_tokens=100_000, max_cost_usd=0.50, max_steps=20):
self.max_tokens = max_tokens
self.max_cost = max_cost_usd
self.max_steps = max_steps
self.used_tokens = 0
self.used_cost = 0
self.steps = 0def track(self, usage, model="gpt-4o"):
self.steps += 1
self.used_tokens += usage.total_tokens
self.used_cost += self.calculate_cost(usage, model)if self.steps > self.max_steps:
raise BudgetExceeded(f"Max steps ({self.max_steps}) exceeded")
if self.used_tokens > self.max_tokens:
raise BudgetExceeded(f"Token budget ({self.max_tokens}) exceeded")
if self.used_cost > self.max_cost:
raise BudgetExceeded(f"Cost budget (${self.max_cost}) exceeded")Production Cost Control Architecture
For real production systems, you need multiple layers:
┌─────────────────────────────────────────────┐
│ Application │
├─────────────────────────────────────────────┤
│ Layer 1: Per-User Budget ($X/day) │
├─────────────────────────────────────────────┤
│ Layer 2: Per-Request Token Limit (max_tokens)│├─────────────────────────────────────────────┤
│ Layer 3: Client-side Rate Limiter │
│ (prevent 429s before they happen) │
├─────────────────────────────────────────────┤
│ Layer 4: Retry with Exponential Backoff │
│ (handle 429s when they do happen) │
├─────────────────────────────────────────────┤
│ Layer 5: Provider-side Rate Limits │
│ (TPM, RPM — enforced by OpenAI) │
├─────────────────────────────────────────────┤
│ Layer 6: Billing Alerts ($50, $100, $500) │
│ (email/Slack when thresholds hit) │
└─────────────────────────────────────────────┘
Quota Management for Teams
When multiple services share one API key:
python
Allocate quotas per service
QUOTAS = {
"chatbot": {"tpm": 80_000, "rpm": 200}, # 53% of total
"search": {"tpm": 40_000, "rpm": 150}, # 27% of total
"agents": {"tpm": 30_000, "rpm": 100}, # 20% of total}
Total: 150,000 TPM, 450 RPM (leaving 50 RPM buffer)
class QuotaManager:
def __init__(self, quotas):
self.limiters = {
service: TokenRateLimiter(q["tpm"], q["rpm"])
for service, q in quotas.items()
}def call(self, service, messages):
limiter = self.limiters[service]
estimated = limiter.estimate_tokens(messages)
limiter.wait_if_needed(estimated)
return client.chat.completions.create(
model="gpt-4o",
messages=messages
)🔬
Model Fallback Under Rate Limits
When your primary model is rate-limited, fall back to alternatives:
python
MODEL_CHAIN = [
{"model": "gpt-4o", "provider": "openai"},
{"model": "claude-sonnet-4-20250514", "provider": "anthropic"},
{"model": "gpt-4o-mini", "provider": "openai"}, # Cheaper, higher limits]
def resilient_call(messages, max_retries=3):
for model_config in MODEL_CHAIN:
for attempt in range(max_retries):
try:
return call_provider(
model_config["provider"],
model_config["model"],
messages
)
except RateLimitError:
if attempt < max_retries - 1:
time.sleep(2 ** attempt)
continue
break # Try next model
raise AllModelsRateLimited("All models exhausted")🛡️
Monitoring and Alerting
python
Track rate limit metrics
import prometheus_client as prom
rate_limit_hits = prom.Counter(
'llm_rate_limit_hits_total',
'Number of 429 responses received',
['model', 'provider'])
tokens_used = prom.Counter(
'llm_tokens_used_total',
'Total tokens consumed',
['model', 'direction'] # direction: input/output)
cost_usd = prom.Counter(
'llm_cost_usd_total',
'Total cost in USD',
['model'])
Alert rules (Prometheus/Grafana)
alert: HighRateLimitRate
expr: rate(llm_rate_limit_hits_total[5m]) > 0.1
for: 5m
summary: "Getting rate limited frequently"
alert: DailyCostHigh
expr: increase(llm_cost_usd_total[24h]) > 50
summary: "Daily LLM spend exceeding $50"
📦
The Cost Optimization Playbook
Strategy Effort Savings
| Use max_tokens to cap output | Low | 10-30% |
|---|---|---|
| Use cheaper models where possible | Low | 50-80% |
| Enable prompt caching (Episode 38) | Low | 50-90% on input |
| Client-side rate limiting | Medium | Prevents 429s |
| Model fallback chains | Medium | Prevents downtime |
| Per-user budgets | Medium | Prevents abuse |
| Batch similar requests | Medium | 20-40% |
| Cache responses for identical queries | High | 80-95% on repeat queries |
🚀
Practical Takeaways
Always implement exponential backoff — it's the minimum viable retry strategy
Read rate limit headers — they tell you exactly how long to wait
Set max_tokens on every request — unbounded output is unbounded cost
Use per-user and per-task budgets — one runaway user shouldn't bankrupt you
Build model fallback chains — when GPT-4o is limited, fall back to alternatives
Monitor and alert on 429s and spend — you can't fix what you can't see
🔄
What's Next?
Episode 40: Load Balancing Multiple Models — Rate limits on one provider? Use two. Three. Ten. Load balancing across multiple models and providers is how production systems stay reliable. Cost-based routing, latency-based routing, and fallback chains.
← Previous
Ep 38: Prompt Caching
Next →
Ep 40: Load Balancing Multiple Models
Next: Episode 40 — Load Balancing Multiple Models
One provider. One model. One point of failure. Production AI needs load balancing across multiple models and providers.
This is part of a 98-episode series covering AI engineering from tokens to production deployment.