Model Gateways: One API to Rule Them All

You're building an AI app. You start with OpenAI. Then a user says Claude is better at coding. Then Gemini drops a million-token context window. Then your boss says "why are we locked into one vendor?"

Now you have three different SDKs, three different auth systems, three different response formats, and three different billing dashboards. Your codebase looks like a Frankenstein monster of if provider == "openai" statements.

Model gateways solve this. One API endpoint. One format. All providers. Swap models with a config change, not a code change.

What Is a Model Gateway?

A model gateway sits between your application and LLM providers. Every request goes through it:

Your App → Model Gateway → OpenAI / Anthropic / Google / Local Models

│
                ├── Unified API format
                ├── Load balancing
                ├── Fallback routing
                ├── Rate limit management
                ├── Cost tracking
                ├── Logging & observability
                └── Caching

Think of it like an API gateway (Kong, Nginx) but specialized for LLM traffic. It understands tokens, streaming, function calling, and the quirks of each provider.

Without a Gateway

python

Without gateway — provider-specific code everywhere

if model.startswith("gpt"):

from openai import OpenAI
    client = OpenAI(api_key=os.environ["OPENAI_KEY"])
    response = client.chat.completions.create(
        model=model, messages=messages
    )
    return response.choices[0].message.content

elif model.startswith("claude"):

import anthropic
    client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_KEY"])
    response = client.messages.create(
        model=model, messages=messages, max_tokens=1024
    )
    return response.content[0].text

elif model.startswith("gemini"):

import google.generativeai as genai
    genai.configure(api_key=os.environ["GOOGLE_KEY"])
    model = genai.GenerativeModel(model)
    response = model.generate_content(messages)
    return response.text

Three SDKs. Three response formats. Three error handling patterns. And this gets worse with every provider you add.

With a Gateway

python

With gateway — one SDK, one format, any provider

from openai import OpenAI

client = OpenAI(
base_url="http://localhost:4000",  # Your gateway
    api_key="your-gateway-key"

)

Switch models by changing one string

response = client.chat.completions.create(
model="anthropic/claude-sonnet-4-5",  # or "gpt-4o" or "gemini-pro"
    messages=messages

)

return response.choices[0].message.content

Same code. Any model. Any provider. The gateway handles all the translation.

LiteLLM: The Open-Source Standard

LiteLLM is the most popular open-source model gateway. It supports 100+ LLM providers through the OpenAI API format.

Setup

bash

pip install litellm[proxy]

Start the proxy server

litellm --model gpt-4o --port 4000

Or with a config file (recommended)

litellm --config config.yaml --port 4000

Configuration

yaml

config.yaml

model_list:

Primary model

• model_name: "default"

litellm_params:
      model: "gpt-4o"
      api_key: "sk-..."

Fallback for the same model name

• model_name: "default"

litellm_params:
      model: "anthropic/claude-sonnet-4-5"
      api_key: "sk-ant-..."

Budget model

• model_name: "fast"

litellm_params:
      model: "gpt-4o-mini"
      api_key: "sk-..."

Local model via Ollama

• model_name: "local"

litellm_params:
      model: "ollama/llama3.1"
      api_base: "http://localhost:11434"

Routing strategy

router_settings:

routing_strategy: "latency-based-routing" # or "cost-based-routing"

num_retries: 3

timeout: 30

fallbacks: [{"default": ["fast"]}] # If "default" fails, try "fast"

Budget limits

general_settings:

master_key: "sk-gateway-master-key"

max_budget: 100 # $100 max spend

budget_duration: "monthly"

Key Features

Feature What It Does

Using LiteLLM as a Library (No Server)

Don't want to run a proxy server? Use it as a Python library:

python

import litellm

Same function, any provider

response = litellm.completion(
model="anthropic/claude-sonnet-4-5",
    messages=[{"role": "user", "content": "Hello!"}]

)

Streaming works the same way

for chunk in litellm.completion(

model="gpt-4o",
    messages=[{"role": "user", "content": "Tell me a story"}],
    stream=True

):

print(chunk.choices[0].delta.content, end="")

🔧

OpenRouter: The Managed Gateway

OpenRouter is a hosted model gateway — you don't run anything. One API key, 200+ models.

How It Works

python

from openai import OpenAI

client = OpenAI(
base_url="https://openrouter.ai/api/v1",
    api_key="sk-or-..."  # One key for all providers

)

Access any model from any provider

response = client.chat.completions.create(
model="anthropic/claude-sonnet-4-5",  # Anthropic
    # model="openai/gpt-4o",            # OpenAI
    # model="google/gemini-pro",         # Google
    # model="meta-llama/llama-3.1-70b",  # Meta (hosted)
    messages=[{"role": "user", "content": "Explain gateways"}]

)

OpenRouter Pricing Model

OpenRouter adds a small markup to provider prices:

Model Direct Price OpenRouter Price Markup

GPT-4o $2.50/$10 per M tokens $2.50/$10 0% (some models)

For hosted open-source models, OpenRouter provides the compute — you pay per token. For commercial models, they often pass through at the same price.

When OpenRouter Shines

Prototyping — Try 200+ models without setting up accounts everywhere

Fallback routing — Auto-route to alternative models on failure

Cost comparison — See real pricing across providers in one dashboard

Access gating — Some models are easier to access through OpenRouter

📊

How Routing Actually Works

The gateway doesn't just proxy requests. It makes intelligent routing decisions.

Routing Strategies

1. Simple Round-Robin

Request 1 → OpenAI

Request 2 → Anthropic

Request 3 → Google

Request 4 → OpenAI

...

Even distribution. Simple. Doesn't account for latency or cost differences.

1. Latency-Based Routing

python

Gateway tracks response times

provider_latency = {
"openai": {"p50": 800, "p95": 2000},      # ms
    "anthropic": {"p50": 600, "p95": 1500},
    "google": {"p50": 1200, "p95": 3000},

}

Routes to the provider with lowest recent latency

Anthropic wins → send request there

Best for user-facing apps where speed matters.

1. Cost-Based Routing

python

Gateway knows pricing

provider_cost = {
"gpt-4o-mini": {"input": 0.15, "output": 0.60},     # per M tokens
    "claude-haiku": {"input": 0.25, "output": 1.25},
    "gemini-flash": {"input": 0.075, "output": 0.30},

}

For non-critical tasks, route to cheapest

Gemini Flash wins → send request there

Best for batch processing and cost-sensitive workloads.

1. Quality-Based Routing

python

Route based on task complexity

def route_by_quality(messages, task_type):

if task_type == "code_generation":
        return "anthropic/claude-sonnet-4-5"  # Best at code
    elif task_type == "creative_writing":
        return "openai/gpt-4o"
    elif task_type == "long_context":
        return "google/gemini-pro"           # 1M context
    elif task_type == "simple_qa":
        return "openai/gpt-4o-mini"          # Cheapest, good enough

Best when you know which models excel at what.

1. Fallback Chains

yaml

If primary fails, try next in chain

fallback_chain:

• model: "gpt-4o"

timeout: 10s

• model: "claude-sonnet-4-5" # If OpenAI is down

timeout: 15s

• model: "gemini-pro" # If both are down

timeout: 20s

• model: "ollama/llama3.1" # Last resort: local model

timeout: 60s

Essential for production. Every major provider has outages.

Building Your Own Gateway (When You Should)

Sometimes LiteLLM and OpenRouter aren't enough. You need a custom gateway when:

You have strict data residency requirements

You need custom routing logic (e.g., user-level model assignments)

You're integrating with internal auth/billing systems

You want to add custom middleware (PII redaction, prompt injection detection)

Minimal Custom Gateway

python

from fastapi import FastAPI, Request

from fastapi.responses import StreamingResponse

import httpx

import json

app = FastAPI()

PROVIDERS = {

"openai": {
        "base_url": "https://api.openai.com/v1",
        "api_key": "sk-..."
    },
    "anthropic": {
        "base_url": "https://api.anthropic.com/v1",
        "api_key": "sk-ant-..."
    }

}

@app.post("/v1/chat/completions")

async def proxy(request: Request):

body = await request.json()
    model = body.get("model", "gpt-4o")

# Route to correct provider
    provider = "anthropic" if "claude" in model else "openai"
    config = PROVIDERS[provider]

# Log the request
    log_request(body, provider)

# Forward to provider
    async with httpx.AsyncClient() as client:
        response = await client.post(
            f"{config['base_url']}/chat/completions",
            json=body,
            headers={"Authorization": f"Bearer {config['api_key']}"},
            timeout=30.0
        )

# Log response and token usage
    result = response.json()
    log_response(result, provider)

return result

This is simplified — a real gateway needs streaming support, error handling, retries, and the Anthropic-to-OpenAI format translation. That's why most people use LiteLLM.

Gateway Comparison

Feature LiteLLM (Self-hosted) OpenRouter (Managed) Custom

🔬

The Unified API Dream

The real value of gateways isn't just convenience. It's optionality:

No vendor lock-in — switch providers without code changes

Cost optimization — route cheap queries to cheap models

Reliability — automatic failover when providers go down

A/B testing — test different models on real traffic

Budget control — hard spend limits per user/team

The AI industry is still young. Pricing changes weekly. New models drop monthly. Providers have outages. A gateway lets you adapt without rewrites.

🛡️

Practical Takeaways

Use a gateway from day one — even if you only use one provider today, you won't tomorrow

LiteLLM for self-hosted — most mature open-source option, great for production

OpenRouter for quick prototyping — one API key, 200+ models, zero setup

Set up fallback chains — every provider has outages, don't let that be your outage

Route by task complexity — don't send simple classification tasks to GPT-4o

Track cost per user/feature — gateways make this easy, and you'll be glad you did

📦

What's Next?

Episode 43: When to Fine-Tune — You've been using APIs and RAG. But what if you need the model to behave differently at a fundamental level? Fine-tuning changes the model's weights — and knowing when (and when not) to do it is one of the most important decisions in AI engineering.

← Previous

Ep 41: Edge Deployment

Next →

Ep 43: When to Fine-Tune vs When to RAG

Next: Episode 43 — When to Fine-Tune vs When to RAG

The model sounds wrong. Doesn't use your terminology, doesn't match your voice. Should you fine-tune or RAG? Wrong answer costs months.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.