Batching & Concurrency: Serving AI to a Thousand Users

Your AI model is running. One user sends a request. Response comes back in 500ms. Beautiful.

Now 100 users send requests at the same time. Without batching, user #100 waits for 99 other requests to finish. That's a 50-second wait. For a chatbot. Nobody's waiting 50 seconds.

Batching is how you go from serving 1 user to serving 1,000 — on the same hardware. It's the single most important technique in production AI serving, and it's the reason cloud AI APIs feel fast even with millions of users.

Why Batching Works: The GPU Utilization Problem

Here's a secret about LLM inference: a single request barely uses the GPU.

From Episode 8, you know GPUs have thousands of cores designed for parallel math. But a single token generation for one user's request is a matrix-vector multiplication — one vector times a matrix. That's like hiring 16,000 workers to carry one box.

Single request:

Computation: [1 × 8192] × [8192 × 8192] = matrix-vector multiply

GPU utilization: ~5-15%

Most CUDA cores: idle

Batch of 32 requests:

Computation: [32 × 8192] × [8192 × 8192] = matrix-matrix multiply

GPU utilization: ~60-80%

CUDA cores: actually working

When you batch 32 requests together, the matrix-vector multiply becomes a matrix-matrix multiply. The GPU can parallelize this much more efficiently. You get 32x the work done in roughly 1.5-2x the time — not 32x the time.

This is the fundamental insight: throughput increases dramatically with batching while per-request latency increases only slightly.

Static Batching: The Naive Approach

The simplest batching: collect N requests, process them together, return all results.

Static batch (size=4):

Request A (100 tokens output) [████████████████████]

Request B (50 tokens output) [██████████░░░░░░░░░░] ← padding

Request C (150 tokens output) [██████████████████████████████]

Request D (30 tokens output) [██████░░░░░░░░░░░░░░░░░░░░░░] ← padding

All requests must wait for the LONGEST one (C = 150 tokens)

Requests B and D waste GPU cycles on padding

Problems:

Short requests wait for long ones — Request D (30 tokens) waits for Request C (150 tokens)

Wasted compute on padding — GPU processes empty tokens for shorter sequences

Blocking — no new requests can start until the entire batch finishes

This is how early inference engines worked. It's how Ollama still works for the most part. It's terrible for production.

🔧

Continuous Batching: The Game Changer

Continuous batching (also called iteration-level batching) fixes everything.

Instead of waiting for a batch to complete, the system operates at the individual token level:

Continuous batching:

Time → t1 t2 t3 t4 t5 t6 t7 t8 t9 t10

Req A: [█] [█] [█] [█] [█] [DONE]

Req B: [█] [█] [█] [█] [█] [█] [█] [█] [DONE]

Req C: [█] [█] [█] [DONE]

Req D: [█] [█] [█] [█] [█] [█] [DONE]

Req E: [█] [█] [█] [█] [█] [DONE]

↑ New requests join immediately when slots open

↑ Completed requests leave immediately — no waiting

At each time step:

Generate ONE token for EACH active request in the batch

If a request is done (hit EOS or max tokens), remove it

If there's a waiting request, add it to the batch immediately

Repeat

No padding. No waiting. No wasted GPU cycles.

How It Works Internally

Iteration 1:

Active requests: [A, B]

Generate: token for A, token for B simultaneously

→ A gets "The", B gets "Hello"

Iteration 2:

Active requests: [A, B, C] ← C just arrived, joins immediately

Generate: token for A, token for B, token for C simultaneously

→ A gets "cat", B gets "world", C gets "First"

Iteration 5:

A finishes! Remove from batch.

D is waiting → joins immediately

Active requests: [B, C, D]

Iteration 7:

C finishes! Remove.

E joins.

Active requests: [B, D, E]

The GPU is ALWAYS processing a full batch. No idle time between requests. This is what makes vLLM and TGI (from Episode 11) so much faster than Ollama for multi-user serving.

📊

Dynamic Batching: The Request Queue

Dynamic batching sits in front of the inference engine. It collects incoming requests and groups them intelligently before forwarding to the model.

Dynamic batching with 50ms window:

t=0ms:    Request A arrives → start timer
t=15ms:   Request B arrives → add to batch
t=35ms:   Request C arrives → add to batch
t=50ms:   Timer expires → send batch [A, B, C] to GPU
t=51ms:   Request D arrives → start new timer
t=70ms:   Request E arrives → add to batch

...

Configuration Knobs

Parameter What It Does Tradeoff

python

Triton Inference Server dynamic batching config

dynamic_batching {

max_queue_delay_microseconds: 50000  # 50ms wait
    preferred_batch_size: [8, 16, 32]    # Target these batch sizes

}

Throughput vs Latency: The Fundamental Tradeoff

This is the core tension in production AI serving:

Throughput (tok/s total)
                      ↑
                      |         ★ sweet spot
                      |       /
                      |     /
                      |   /
                      | /
                      +------------------→
                         Latency per request

Metric Optimize For Who Cares

Real Numbers

On an A100 80GB with Llama 3.1 8B:

Look at the pattern:

Throughput increases 27x from batch 1 to 128

Per-user latency drops from 95 to 20 tok/s

Sweet spot is around batch 32-64 — high throughput, acceptable latency

At batch=1, the A100 is 92% idle. You're paying $32/hour for a GPU that's mostly sleeping.

Prefill vs Decode: Two Different Bottlenecks

LLM inference has two distinct phases, and they batch differently:

Prefill Phase (Processing the Prompt)

All input tokens are processed in parallel. This is compute-bound:

Prompt: "Write a Python function that sorts a list"

Tokens: [Write, a, Python, function, that, sorts, a, list]

All 8 tokens processed simultaneously in one forward pass

→ Fast, GPU-efficient, benefits enormously from batching

Decode Phase (Generating Output)

Tokens generated one at a time. This is memory-bandwidth-bound (KV cache reads):

Generation:

Step 1: Produce "def" → read KV cache (8 entries)

Step 2: Produce "sort" → read KV cache (9 entries)

Step 3: Produce "_list" → read KV cache (10 entries)

...

The Scheduling Problem

Prefill is fast and compute-heavy. Decode is slow and memory-heavy. Mixing them in one batch creates conflicts:

Bad scheduling:

Prefill for Request X (100 prompt tokens) → 50ms, blocks GPU
Decode for Requests A,B,C (already mid-generation) → STALLED for 50ms
Users A,B,C see a 50ms pause mid-response → feels janky

Chunked prefill solves this by breaking long prompts into chunks:

Chunked prefill:

Prefill chunk 1 of X (25 tokens) → 12ms
Decode for A,B,C → 3ms
Prefill chunk 2 of X (25 tokens) → 12ms
Decode for A,B,C → 3ms

...

Users A,B,C see smooth generation

User X's prefill takes slightly longer but doesn't block others

vLLM supports this with --enable-chunked-prefill.

🔬

Concurrency in Practice

Single GPU, Multiple Users

bash

vLLM with optimized concurrency settings

vllm serve meta-llama/Llama-3.1-8B-Instruct \

--max-num-seqs 64 \ # Max concurrent requests

--max-num-batched-tokens 8192 \ # Total tokens per batch step

--enable-chunked-prefill \     # Don't let prefill block decode

--gpu-memory-utilization 0.9 # Use 90% of VRAM

Multiple GPUs, High Traffic

bash

Tensor parallelism: split model across GPUs

vllm serve meta-llama/Llama-3.1-70B-Instruct \

--tensor-parallel-size 4 \ # 4 GPUs for one model

--max-num-seqs 128 \

--enable-chunked-prefill

Or pipeline parallelism: each GPU handles different layers

Useful when GPU interconnect is slow

Load Balancing Across Instances

For really high traffic, run multiple vLLM instances behind a load balancer:

Users → Load Balancer → vLLM Instance 1 (4× A100)

→ vLLM Instance 2 (4× A100)
                      → vLLM Instance 3 (4× A100)

nginx

nginx load balancer config

upstream vllm_cluster {

least_conn;  # Send to the instance with fewest connections
    server vllm1:8000;
    server vllm2:8000;
    server vllm3:8000;

}

🛡️

Measuring What Matters

Key metrics for production AI serving:

python

The metrics you should monitor:

1. Time to First Token (TTFT)

How long until the user sees the first word

Target: < 500ms for interactive apps

2. Inter-Token Latency (ITL)

Time between consecutive tokens

Target: < 50ms (feels like smooth typing)

3. Throughput (tokens/second/GPU)

Total tokens generated across all users

Target: maximize while keeping TTFT < 500ms

4. Queue depth

Number of requests waiting to be processed

Target: < 10 (if consistently higher, add capacity)

5. GPU memory utilization

How full is VRAM (model + KV caches)

Target: 80-90% (too low = wasting money, too high = OOM risk)

📦

Try It Yourself — Benchmark Throughput

bash

Using vLLM's built-in benchmark

python -m vllm.entrypoints.openai.api_server \

--model meta-llama/Llama-3.1-8B-Instruct &

Run benchmark with increasing concurrency

python benchmarks/benchmark_serving.py \

--model meta-llama/Llama-3.1-8B-Instruct \

--num-prompts 100 \

--request-rate 10 \ # 10 requests per second

--backend openai

Compare with different batch sizes

for batch in 1 4 16 32 64; do

echo "=== Batch size: $batch ==="

python benchmarks/benchmark_throughput.py \

--model meta-llama/Llama-3.1-8B-Instruct \
    --num-prompts $batch \
    --output-len 128

done

🚀

Practical Takeaways

A single request uses < 15% of GPU capacity — batching is how you unlock the other 85%

Continuous batching is non-negotiable for production — no padding, no waiting, no wasted compute

Throughput scales 20-30x from batch=1 to batch=64 — while per-user latency only drops ~2x

Prefill and decode have different bottlenecks — use chunked prefill to prevent decode stalls

Monitor TTFT, ITL, and throughput — these three numbers tell you everything about serving quality

The sweet spot is 32-64 concurrent requests — beyond that, per-user latency starts hurting

🔄

What's Next?

Episode 14: Speculative Decoding — What if you could predict multiple tokens at once instead of one at a time? Use a small fast model to "guess" and a large smart model to "verify." It's like getting a junior dev to write a first draft and a senior to approve it. 2-3x speed gains with zero quality loss.

← Previous

Ep 12: KV Cache

Next →

Ep 14: Speculative Decoding

Next: Episode 14 — Speculative Decoding

Here's a management trick that works for both humans and AI: let a junior do the draft, let the senior just approve or reject.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.