Streaming: Why AI Responds One Word at a Time (And Why It Feels Fast)

Open ChatGPT. Ask it anything. Watch the words appear one by one, like someone typing in real time.

That's not a UI animation. It's not a deliberate design choice to make it "feel" more human. The model genuinely doesn't know word 2 until word 1 is generated. Each token is a fresh prediction, and streaming delivers each one the instant it's ready.

Without streaming, you'd stare at a blank screen for 5-15 seconds, then get the entire answer dumped at once. With streaming, you start reading within 200 milliseconds. Same total time, completely different experience.

Here's how it works.

Why Streaming Exists

LLMs generate tokens sequentially. A 500-token response takes about 5-10 seconds (depending on the model and provider). Without streaming:

User sends message → [5 seconds of nothing] → Entire response appears

With streaming:

User sends message → [200ms] → "The" → "capital" → "of" → "France" → "is" → "Paris."

The total time is the same. But the time to first token (TTFT) drops from 5 seconds to ~200ms. The user starts reading immediately, and the response feels instant.

Metric Without Streaming With Streaming

Server-Sent Events (SSE)

Streaming uses Server-Sent Events (SSE) — a simple HTTP protocol for server-to-client streaming. It's just regular HTTP with a special content type:

HTTP/1.1 200 OK

Content-Type: text/event-stream

Cache-Control: no-cache

Connection: keep-alive

data: {"id":"chatcmpl-123","choices":[{"delta":{"content":"The"}}]}

data: {"id":"chatcmpl-123","choices":[{"delta":{"content":" capital"}}]}

data: {"id":"chatcmpl-123","choices":[{"delta":{"content":" of"}}]}

data: {"id":"chatcmpl-123","choices":[{"delta":{"content":" France"}}]}

data: {"id":"chatcmpl-123","choices":[{"delta":{"content":" is"}}]}

data: {"id":"chatcmpl-123","choices":[{"delta":{"content":" Paris"}}]}

data: {"id":"chatcmpl-123","choices":[{"delta":{"content":"."}}]}

data: {"id":"chatcmpl-123","choices":[{"finish_reason":"stop"}]}

data: [DONE]

Each line starting with data: is a chunk. Each chunk contains a delta — the new token(s) being added. Your client reads these chunks as they arrive and appends them to the display.

Key Differences from Regular HTTP

Regular HTTP SSE (Streaming)

🔧

Implementing Streaming

Python (OpenAI SDK)

python

from openai import OpenAI

client = OpenAI()

Non-streaming (wait for full response)

response = client.chat.completions.create(
model="gpt-4o",
    messages=[{"role": "user", "content": "Write a haiku about coding"}],
    stream=False  # Default

)

print(response.choices[0].message.content)
Streaming (get tokens as they generate)

stream = client.chat.completions.create(
model="gpt-4o",
    messages=[{"role": "user", "content": "Write a haiku about coding"}],
    stream=True

)

full_response = ""

for chunk in stream:

if chunk.choices[0].delta.content:
        token = chunk.choices[0].delta.content
        print(token, end="", flush=True)  # Print without newline
        full_response += token

print()  # Final newline

JavaScript/TypeScript (Frontend)

javascript

async function streamChat(message) {

const response = await fetch('https://api.openai.com/v1/chat/completions', {
        method: 'POST',
        headers: {
            'Authorization': `Bearer ${API_KEY}`,
            'Content-Type': 'application/json',
        },
        body: JSON.stringify({
            model: 'gpt-4o',
            messages: [{ role: 'user', content: message }],
            stream: true,
        }),
    });

const reader = response.body.getReader();
    const decoder = new TextDecoder();
    let buffer = '';

while (true) {
        const { done, value } = await reader.read();
        if (done) break;

buffer += decoder.decode(value, { stream: true });
        const lines = buffer.split('\n');
        buffer = lines.pop(); // Keep incomplete line in buffer

for (const line of lines) {
            if (line.startsWith('data: ')) {
                const data = line.slice(6);
                if (data === '[DONE]') return;

const parsed = JSON.parse(data);
                const content = parsed.choices[0]?.delta?.content;
                if (content) {
                    document.getElementById('output').textContent += content;
                }
            }
        }
    }

}

Node.js Backend (Proxy Streaming to Client)

javascript

// Express server that proxies streaming from OpenAI to the browser

app.post('/api/chat', async (req, res) => {
// Set SSE headers
    res.setHeader('Content-Type', 'text/event-stream');
    res.setHeader('Cache-Control', 'no-cache');
    res.setHeader('Connection', 'keep-alive');
const stream = await openai.chat.completions.create({
        model: 'gpt-4o',
        messages: req.body.messages,
        stream: true,
    });
for await (const chunk of stream) {
        const content = chunk.choices[0]?.delta?.content;
        if (content) {
            res.write(`data: ${JSON.stringify({ content })}\n\n`);
        }
    }
res.write('data: [DONE]\n\n');
    res.end();

});

📊

The Streaming Response Format

Non-streaming and streaming responses have different structures:

Non-Streaming Response

json

{

"choices": [{
        "message": {
            "role": "assistant",
            "content": "The capital of France is Paris."
        },
        "finish_reason": "stop"
    }],
    "usage": {
        "prompt_tokens": 15,
        "completion_tokens": 8,
        "total_tokens": 23
    }

}

Streaming Chunks

json

// First chunk (role)

{"choices": [{"delta": {"role": "assistant", "content": ""}}]}

// Content chunks

{"choices": [{"delta": {"content": "The"}}]}

{"choices": [{"delta": {"content": " capital"}}]}

{"choices": [{"delta": {"content": " of"}}]}

// Final chunk

{"choices": [{"delta": {}}, "finish_reason": "stop"]}

// Usage (if stream_options.include_usage=true)

{"usage": {"prompt_tokens": 15, "completion_tokens": 8}}

Notice: streaming uses delta instead of message. Each delta contains only the new content, not the full message so far. You assemble the full message client-side.

Streaming with Tool Calls

Tool calls stream differently — instead of content tokens, you get function call fragments:

json

// Tool call starts

{"choices": [{"delta": {"tool_calls": [{"index": 0, "id": "call_abc", "function": {"name": "get_weather", "arguments": ""}}]}}]}

// Arguments stream in chunks

{"choices": [{"delta": {"tool_calls": [{"index": 0, "function": {"arguments": "{\"ci"}}]}}]}

{"choices": [{"delta": {"tool_calls": [{"index": 0, "function": {"arguments": "ty\":"}}]}}]}

{"choices": [{"delta": {"tool_calls": [{"index": 0, "function": {"arguments": " \"Mum"}}]}}]}

{"choices": [{"delta": {"tool_calls": [{"index": 0, "function": {"arguments": "bai\"}"}}]}}]}

You need to accumulate the argument fragments and parse the complete JSON once streaming finishes:

python

tool_calls = {}

for chunk in stream:

delta = chunk.choices[0].delta

if delta.tool_calls:
        for tc in delta.tool_calls:
            idx = tc.index
            if idx not in tool_calls:
                tool_calls[idx] = {"id": tc.id, "name": tc.function.name, "args": ""}
            if tc.function.arguments:
                tool_calls[idx]["args"] += tc.function.arguments

After streaming completes:

for tc in tool_calls.values():

args = json.loads(tc["args"])  # Now parse complete JSON
    result = execute_tool(tc["name"], args)

Why Streaming Feels Faster (Psychology)

The actual total time is the same or even slightly longer with streaming (SSE overhead). But it feels 3-5x faster because of how human perception works:

1. Progress indicator effect. Seeing tokens appear tells your brain "it's working." A blank screen creates uncertainty — is it broken? Is it thinking? Is it slow?

1. Read-while-generating. A 500-token response takes 5 seconds to generate. Without streaming, you wait 5 seconds then start reading (total: 5 + reading time). With streaming, you start reading at 200ms. By the time the last token arrives, you've already read most of the response.

1. Engagement bias. Watching text appear is more engaging than waiting. Engaged time feels shorter than idle time. Same reason a cricket match feels shorter than waiting in a queue.

🔬

When NOT to Stream

Streaming isn't always the right choice:

Scenario Stream? Why

🛡️

Chunked Transfer Encoding

Under the hood, SSE uses chunked transfer encoding — an HTTP/1.1 feature that allows sending data in pieces without knowing the total size upfront:

HTTP/1.1 200 OK

Transfer-Encoding: chunked

1a\r\n

data: {"content":"Hello"}\r\n

\r\n

1c\r\n

data: {"content":" world"}\r\n

\r\n

0\r\n

\r\n

Each chunk is prefixed with its size in hexadecimal. The final chunk has size 0, signaling the end. Most HTTP libraries handle this transparently — you just read the stream.

📦

Performance Numbers

Real-world streaming performance for major providers:

TTFT matters most for user experience. A model that starts responding in 100ms feels instant, even if total generation takes 10 seconds.

🚀

Practical Takeaways

Always stream for user-facing interfaces — the UX difference is dramatic

Skip streaming for backend processing — simpler code, same total time

TTFT matters more than total speed — users judge by how fast the first word appears

Accumulate tool call arguments — they stream in fragments, parse after completion

SSE is just HTTP with a special content type — no WebSocket, no special protocol

Streaming makes the same speed feel 3-5x faster — it's psychology, not physics

🔄

What's Next?

Episode 38: Prompt Caching — What if the API could remember your system prompt and skip re-processing it every time? Anthropic and OpenAI both offer prompt caching, and it can cut your costs by 90%. How it works, when it kicks in, and how to design prompts for maximum cache hits.

← Previous

Ep 36: The OpenAI API Spec

Next →

Ep 38: Prompt Caching

Next: Episode 38 — Prompt Caching

Every API call re-reads your entire system prompt. Every. Single. Time. Prompt caching fixes this — and slashes your bill.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.