Your app runs on GPT-4o. One provider. One model. One point of failure.
OpenAI goes down (it happens more than you'd think), and your entire application dies. Or worse — OpenAI doesn't go down, but you hit your rate limit, and 40% of your requests start failing during peak hours.
Production AI systems don't rely on a single model or provider. They load balance across multiple options — routing requests based on cost, latency, availability, and capability. Same app, multiple brains, zero downtime.
Here's how to build it.
Why Load Balance?
Problem Single Provider Multi-Provider
Provider outage App is dead Traffic shifts to backup
Rate limits Requests fail Overflow to alternative
Cost spikes Stuck with one price Route to cheapest option
Latency spikes Users wait Route to fastest available
Model quality issues No alternative Switch models instantly
Vendor lock-in Completely locked Switch providers anytime
It's the same principle as running multiple servers behind NGINX. You don't serve your website from one machine. Don't serve AI from one provider.
Routing Strategies
- Round Robin (Simplest)
Distribute requests evenly across providers:
python
import itertools
providers = itertools.cycle([
{"name": "openai", "model": "gpt-4o", "client": openai_client},
{"name": "anthropic", "model": "claude-sonnet-4-20250514", "client": anthropic_client},
{"name": "groq", "model": "llama-3.1-70b", "client": groq_client},])
def route_request(messages):
provider = next(providers)
return call_provider(provider, messages)Pros: Dead simple. Even distribution.
Cons: Ignores cost, latency, and capability differences. Treats all providers equally when they aren't.
- Cost-Based Routing
Route to the cheapest provider that can handle the request:
python
PROVIDER_COSTS = {
"gpt-4o": {"input": 2.50, "output": 10.00},
"claude-sonnet": {"input": 3.00, "output": 15.00},
"gpt-4o-mini": {"input": 0.15, "output": 0.60},
"llama-3.1-70b": {"input": 0.59, "output": 0.79}, # via Groq
"gemini-flash": {"input": 0.075, "output": 0.30},}
def route_by_cost(messages, required_quality="high"):
if required_quality == "high":
candidates = ["gpt-4o", "claude-sonnet"]
elif required_quality == "medium":
candidates = ["gpt-4o-mini", "llama-3.1-70b"]
else:
candidates = ["gemini-flash", "gpt-4o-mini"]
# Sort by cost (input + output weighted)
candidates.sort(key=lambda m: PROVIDER_COSTS[m]["output"])
for model in candidates:
if is_available(model):
return call_model(model, messages)
raise NoModelAvailable("All candidates rate-limited or down")Real savings example:
| Request Type | Single Provider (GPT-4o) | Cost-Based Routing | Savings |
|---|---|---|---|
| Simple chat | $0.01 | $0.001 (Mini) | 90% |
| Code review | $0.05 | $0.005 (Llama) | 90% |
| Complex reasoning | $0.05 | $0.05 (GPT-4o) | 0% |
| Daily total (1000 requests) | $37 | $8 | 78% |
Most requests don't need the most expensive model. Route them to cheaper alternatives.
- Latency-Based Routing
Route to the fastest available provider:
python
import time
from collections import defaultdict
Track rolling average latency
latency_history = defaultdict(list)
def track_latency(provider, latency_ms):
latency_history[provider].append(latency_ms)
# Keep last 100 measurements
if len(latency_history[provider]) > 100:
latency_history[provider] = latency_history[provider][-100:]def avg_latency(provider):
history = latency_history.get(provider, [1000]) # Default 1s
return sum(history) / len(history)def route_by_latency(messages, candidates=None):
candidates = candidates or ["openai", "anthropic", "groq"]
# Sort by recent average latency
candidates.sort(key=lambda p: avg_latency(p))
for provider in candidates:
start = time.time()
try:
result = call_provider(provider, messages)
track_latency(provider, (time.time() - start) * 1000)
return result
except Exception:
track_latency(provider, 30000) # Mark as slow on failure
continue- Capability-Based Routing
Different models excel at different tasks:
python
CAPABILITY_MAP = {
"code_generation": ["claude-sonnet", "gpt-4o", "deepseek-coder"],
"creative_writing": ["gpt-4o", "claude-sonnet"],
"data_extraction": ["gpt-4o-mini", "gemini-flash"], # Cheaper is fine
"math_reasoning": ["gpt-4o", "claude-sonnet"],
"translation": ["gpt-4o-mini", "gemini-flash"],
"summarization": ["gemini-flash", "gpt-4o-mini", "llama-3.1-70b"],
"vision": ["gpt-4o", "claude-sonnet", "gemini-pro"],}
def route_by_capability(messages, task_type):
candidates = CAPABILITY_MAP.get(task_type, ["gpt-4o"])for model in candidates:
if is_available(model):
return call_model(model, messages)- Weighted Distribution
Give more traffic to preferred providers:
python
import random
WEIGHTS = {
"openai": 50, # 50% of traffic
"anthropic": 30, # 30% of traffic
"groq": 20, # 20% of traffic}
def route_weighted(messages):
providers = list(WEIGHTS.keys())
weights = list(WEIGHTS.values())selected = random.choices(providers, weights=weights, k=1)[0]
return call_provider(selected, messages)🔧
Fallback Chains
The most important pattern. When the primary fails, fall back:
python
class FallbackRouter:
def __init__(self):
self.chains = {
"default": [
{"provider": "openai", "model": "gpt-4o"},
{"provider": "anthropic", "model": "claude-sonnet-4-20250514"},
{"provider": "groq", "model": "llama-3.1-70b-versatile"},
],
"fast": [
{"provider": "groq", "model": "llama-3.1-70b-versatile"},
{"provider": "openai", "model": "gpt-4o-mini"},
{"provider": "anthropic", "model": "claude-haiku-4-5-20241022"},
],
"cheap": [
{"provider": "openai", "model": "gpt-4o-mini"},
{"provider": "google", "model": "gemini-2.0-flash"},
{"provider": "groq", "model": "llama-3.1-8b-instant"},
]
}def call(self, messages, chain="default", max_retries=2):
models = self.chains[chain]
last_error = Nonefor model_config in models:
for attempt in range(max_retries):
try:
return self._call_model(model_config, messages)
except RateLimitError:
time.sleep(2 ** attempt)
continue
except (APIError, Timeout) as e:
last_error = e
break # Try next modelprint(f"Failed {model_config['model']}, trying next...")raise AllModelsExhausted(f"All models failed. Last error: {last_error}")📊
Health Checking
Monitor provider health and route around problems:
python
class ProviderHealth:
def __init__(self):
self.status = {} # provider → {"healthy": bool, "last_check": timestamp}
self.error_counts = defaultdict(int)
self.circuit_breakers = {}def record_success(self, provider):
self.error_counts[provider] = 0
self.status[provider] = {"healthy": True, "last_check": time.time()}def record_failure(self, provider):
self.error_counts[provider] += 1# Circuit breaker: 5 failures → mark unhealthy for 60 seconds
if self.error_counts[provider] >= 5:
self.status[provider] = {"healthy": False, "last_check": time.time()}
self.circuit_breakers[provider] = time.time() + 60
print(f"⚠️ {provider} circuit breaker OPEN (60s cooldown)")def is_healthy(self, provider):
# Check circuit breaker
if provider in self.circuit_breakers:
if time.time() < self.circuit_breakers[provider]:
return False
else:
del self.circuit_breakers[provider] # Cooldown expiredreturn self.status.get(provider, {}).get("healthy", True)health = ProviderHealth()def smart_route(messages, candidates):
healthy = [p for p in candidates if health.is_healthy(p)]if not healthy:
# All providers down — try anyway with the first one
healthy = candidates[:1]for provider in healthy:
try:
result = call_provider(provider, messages)
health.record_success(provider)
return result
except Exception as e:
health.record_failure(provider)Building a Complete Load Balancer
Putting it all together:
python
class AILoadBalancer:
def __init__(self, config):
self.health = ProviderHealth()
self.latency = LatencyTracker()
self.budget = BudgetTracker()
self.config = configdef route(self, messages, **kwargs):
strategy = kwargs.get("strategy", "balanced")
task_type = kwargs.get("task_type", "general")
max_cost = kwargs.get("max_cost_usd", 0.10)# Get candidates based on strategy
candidates = self._get_candidates(strategy, task_type)# Filter by health
candidates = [c for c in candidates if self.health.is_healthy(c["provider"])]# Filter by budget
candidates = [c for c in candidates
if self._estimated_cost(c, messages) <= max_cost]# Sort by strategy preference
if strategy == "fast":
candidates.sort(key=lambda c: self.latency.avg(c["provider"]))
elif strategy == "cheap":
candidates.sort(key=lambda c: self._estimated_cost(c, messages))# Try each candidate with fallback
for candidate in candidates:
try:
start = time.time()
result = self._call(candidate, messages)
self.latency.track(candidate["provider"], time.time() - start)
self.health.record_success(candidate["provider"])
self.budget.track(candidate, result.usage)
return result
except Exception as e:
self.health.record_failure(candidate["provider"])
continueraise NoModelAvailable("All candidates exhausted")Usage
lb = AILoadBalancer(config)Route by cost for simple requests
result = lb.route(messages, strategy="cheap", task_type="summarization")Route by latency for user-facing chat
result = lb.route(messages, strategy="fast", task_type="chat")Route by quality for complex reasoning
result = lb.route(messages, strategy="quality", task_type="code_generation")Existing Tools
You don't have to build this from scratch:
Tool What It Does Open Source
| LiteLLM | Unified API + load balancing + fallbacks | Yes |
|---|---|---|
| OpenRouter | Hosted routing across 200+ models | No (SaaS) |
| Portkey | AI gateway with routing, caching, logging | Partially |
| Martian | Smart model routing based on prompt | No (SaaS) |
| Not Diamond | ML-based model selection | No (SaaS) |
We'll cover LiteLLM and OpenRouter in detail in Episode 42 (Model Gateways).
🔬
Practical Takeaways
Never rely on a single provider — outages happen, rate limits hit, prices change
Cost-based routing saves 50-80% — most requests don't need the most expensive model
Fallback chains are non-negotiable — primary → secondary → tertiary, always
Health checks prevent cascading failures — circuit breakers protect your app
Track latency per provider — route users to the fastest available option
Use existing tools — LiteLLM and OpenRouter handle most of this for you
🛡️
What's Next?
Episode 41: Edge Deployment — What if you didn't need the cloud at all? Running models on phones, Raspberry Pis, and browsers. On-device inference is the frontier of AI deployment, and it's more practical than you think.
← Previous
Ep 39: Rate Limiting & Token Budgets
Next →
Ep 41: Edge Deployment
Next: Episode 41 — Edge Deployment
Your words travel to Iowa, hit an H100 GPU, and come back. 500ms to 3 seconds. What if the AI ran on your device instead?
This is part of a 98-episode series covering AI engineering from tokens to production deployment.