How ChatGPT Works: The Full Stack, End to End

You type "Explain quantum computing like I'm 5" into ChatGPT. Two seconds later, words start streaming onto your screen.

In those two seconds, your message traveled through a global CDN, hit a load balancer, got routed to an authentication service, was tokenized, passed through a model with trillions of parameters running on a cluster of GPUs, generated tokens one at a time, and streamed back to your browser over a WebSocket connection.

And that's the simple version. The full stack is one of the most complex software architectures ever deployed. Let's trace every step.

The Architecture Overview

Your Browser

│
    ▼

CloudFlare CDN (DDoS protection, caching, edge routing)

│
    ▼

API Gateway (rate limiting, auth, routing)

│
    ▼

Conversation Service (context management, history)

│
    ▼

Safety/Moderation Layer (content filtering, PII detection)

│
    ▼

Prompt Orchestration (system prompts, tool configs, model selection)

│
    ▼

Inference Router (load balancing across GPU clusters)

│
    ▼

Inference Cluster (GPU nodes running the actual model)

│
    ▼

Token Streaming (SSE/WebSocket back to browser)

│
    ▼

Your Screen

Let's walk through each layer.

Layer 1: The Edge (CloudFlare + CDN)

Before your message even reaches OpenAI's servers, it hits CloudFlare — a massive CDN and security layer.

What happens here:

DDoS protection — millions of requests/second during peak, many of them malicious

Geographic routing — connects you to the nearest edge server

TLS termination — your HTTPS connection is decrypted here

Rate limiting (first pass) — obvious abuse gets blocked immediately

WebSocket upgrade — your connection is upgraded from HTTP to WebSocket for streaming

OpenAI handles roughly 100 million weekly active users. Without the CDN layer, their servers would melt.

🔧

Layer 2: API Gateway

The gateway handles authentication and routing:

Request arrives:

{

"model": "gpt-4o",

"messages": [

{"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain quantum computing like I'm 5"}

],

"stream": true

}

What happens here:

Authentication — validate API key or session token

Rate limiting — per-user token limits (TPM, RPM)

Usage tracking — count tokens for billing

Model routing — send GPT-4o requests to GPT-4o cluster, GPT-3.5 to a different one

Feature flags — enable/disable features per user tier (Plus, Team, Enterprise)

For ChatGPT (the consumer product), this also handles:

Session management

Conversation persistence (your chat history)

Plugin/tool activation

Memory features (if enabled)

📊

Layer 3: Safety and Moderation

BEFORE the model sees your prompt, it goes through content moderation:

python

Simplified moderation pipeline

def moderate_input(messages):

# Step 1: Fast classifier (lightweight model)
    risk_score = safety_classifier.predict(messages)

if risk_score > 0.95:
        return BLOCK  # Obviously harmful, don't even process

# Step 2: PII detection
    pii_found = detect_pii(messages)
    if pii_found:
        messages = redact_pii(messages)  # Strip sensitive info

# Step 3: Injection detection
    injection_score = detect_prompt_injection(messages)
    if injection_score > 0.8:
        flag_for_review(messages)
        # Still process but add safety system prompt

return ALLOW

OpenAI uses a combination of:

Lightweight classifiers for fast filtering (sub-millisecond)

The model itself as a secondary safety check (the system prompt includes safety instructions)

Output filtering — even after generation, responses are checked before being sent to the user

This is why ChatGPT sometimes refuses to answer mid-response. The output filter caught something.

Layer 4: Prompt Orchestration

Your simple message gets wrapped in a much larger prompt:

[SYSTEM]: You are ChatGPT, a large language model trained by OpenAI.

Knowledge cutoff: April 2024. Current date: [today].

You have access to the following tools: [browser, code_interpreter, dalle]

When the user asks for images, use DALL-E.

When code execution is needed, use the code interpreter.

[SAFETY INSTRUCTIONS: Do not generate harmful content...]

[MEMORY: User prefers concise answers, works in software engineering...]

[CONVERSATION HISTORY]:

User (3 messages ago): "What's quantum computing?"

Assistant: "Quantum computing uses quantum mechanical phenomena..."

User (1 message ago): "That's too complex"

[CURRENT MESSAGE]:

User: "Explain quantum computing like I'm 5"

What started as a 7-word message is now potentially thousands of tokens, including:

System prompt with personality, safety rules, and tool configs

User memories and preferences

Full conversation history (up to context limit)

Tool definitions in function-calling format

This is why ChatGPT remembers your earlier messages — it's not actually "remembering." It's re-reading the entire conversation every time.

Layer 5: Inference Router

The orchestrated prompt needs to get to a GPU. But which one?

OpenAI operates thousands of GPU nodes across multiple data centers. The inference router decides where to send each request:

Routing decisions:

├── Which data center? (closest with capacity)

├── Which GPU cluster? (model-specific: GPT-4o cluster vs GPT-3.5 cluster)

├── Which node in the cluster? (least loaded)

├── Priority level? (Plus users > free tier)

└── Batch or individual? (can this request be batched with others?)

Key optimization: Request batching. Instead of processing one request at a time, the inference server batches multiple requests together:

Without batching:

Request 1 → GPU → Response 1 (utilization: 30%)

Request 2 → GPU → Response 2 (utilization: 30%)

With batching (continuous):

[Request 1 + Request 2 + Request 3] → GPU → [Response 1, 2, 3]

(utilization: 90%)

This is continuous batching (from Episode 37). The GPU processes multiple requests simultaneously, dynamically adding new requests as others finish. It's what makes serving 100M users economically feasible.

🔬

Layer 6: The Inference Engine

This is where the magic actually happens. A GPU cluster running vLLM, TensorRT-LLM, or OpenAI's custom inference engine processes the prompt.

The Prefill Phase (Processing Your Input)

Input: ~2000 tokens (your message + system prompt + history)

All 2000 tokens processed in PARALLEL on the GPU

→ One forward pass through all transformer layers

→ KV cache populated for every token

→ Ready to generate

Time: ~200ms for GPT-4o

This is fast because all input tokens are processed at once. The GPU loves parallel work.

The Decode Phase (Generating the Response)

Token 1: "Imagine" → 200ms + context reading
Token 2: "you"     → 20ms (faster now, KV cache warm)
Token 3: "have"    → 20ms
Token 4: "a"       → 20ms

...

Token N: "[done]"  → 20ms

Total: First token in ~200ms, then ~50 tokens/second

Each token is generated sequentially. The model:

Reads all previous tokens (using KV cache — no recomputation)

Predicts the next token probability distribution

Samples from the distribution (using temperature, top-p — Episode 6)

Appends the new token

Repeats

The KV cache (Episode 12) is critical here. Without it, the model would re-process all previous tokens for each new token. With it, only the new token requires a full computation pass.

GPU Memory Layout During Inference

GPU Memory (80GB A100):

┌────────────────────────────────────┐

│ Model Weights ~40GB │ (loaded once, shared across requests)

├────────────────────────────────────┤

│ KV Cache (Batch) ~30GB │ (grows with context length × batch size)

├────────────────────────────────────┤

│ Activations ~5GB │ (temporary, per forward pass)

├────────────────────────────────────┤

│ Overhead ~5GB │ (CUDA, framework, buffers)

└────────────────────────────────────┘

For large models like GPT-4 (rumored ~1.8T parameters with MoE), the model is split across multiple GPUs using tensor parallelism and pipeline parallelism.

🛡️

Layer 7: Token Streaming

As each token is generated, it's streamed back to your browser immediately. This is Server-Sent Events (SSE):

HTTP Response (streaming):

data: {"choices":[{"delta":{"content":"Imagine"}}]}

data: {"choices":[{"delta":{"content":" you"}}]}

data: {"choices":[{"delta":{"content":" have"}}]}

data: {"choices":[{"delta":{"content":" a"}}]}

data: {"choices":[{"delta":{"content":" box"}}]}

...

data: [DONE]

The browser receives each chunk and appends it to the UI. This is why you see ChatGPT "typing" — it genuinely doesn't know the next word until it generates it.

Why streaming matters:

Perceived latency drops dramatically. Time-to-first-token is ~200ms. Time-to-full-response might be 10 seconds. Without streaming, users stare at a blank screen for 10 seconds.

Users can read while it generates. By the time the model finishes, the user has already read the first half.

Early termination. If the response starts going off track, the user (or the system) can abort early, saving compute.

📦

The Numbers at Scale

Metric Estimated Value

The economics are tight. Free-tier users cost more per query than they generate. Plus users subsidize the free tier. API customers are the profit center.

🚀

What Makes ChatGPT Feel Fast

OpenAI employs several tricks to minimize latency:

Speculative decoding (Episode 14) — a small draft model generates candidates, the large model verifies in parallel

KV cache optimization — PagedAttention (vLLM) prevents memory waste

Quantization — INT8/INT4 models on some tiers to speed up inference

Geographic routing — process requests in the nearest data center

Continuous batching — never let GPUs idle

Model-specific optimizations — custom CUDA kernels for attention

🔄

Practical Takeaways

ChatGPT isn't one model — it's a distributed system with 7+ layers between your keyboard and the response

Your 7-word prompt becomes thousands of tokens — system prompts, history, safety instructions, tool configs

Safety happens BEFORE and AFTER generation — both input and output are filtered

Streaming isn't a UI trick — it's how inference works — tokens are genuinely generated one at a time

The KV cache is what makes long conversations possible — without it, latency would grow quadratically

The economics are brutal — GPU costs are enormous, and most users don't pay enough to cover their usage

🏗️

What's Next?

Episode 93: How Cursor/Copilot Works — AI coding assistants use a different trick: fill-in-the-middle. Instead of generating left-to-right, they predict what goes IN BETWEEN your existing code. Plus: how they gather context from your entire codebase, integrate with your editor's language server, and make suggestions that actually compile.

← Previous

Ep 91: Evaluation Frameworks

Next →

Ep 93: How Cursor/Copilot Works

Next: Episode 93 — How Cursor/Copilot Works

You pause for half a second. Grey text appears — completing your function, writing the next 5 lines. Here's how it actually works.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.