Ollama vs vLLM vs TGI: Choosing Your Inference Engine

You can run AI locally. Great. Episode 10 showed you how llama.cpp works.

But now your boss says: "Put this in production. Serve 100 users simultaneously. Make it fast. Make it reliable."

llama.cpp alone won't cut it. You need an inference engine — software built specifically for serving LLMs at scale.

Three dominate the landscape: Ollama for local development, vLLM for production GPU serving, and TGI for the Hugging Face ecosystem. They solve different problems. Choosing the wrong one costs you either money or sanity.

The Three Contenders

Ollama vLLM TGI

All three are open source. All three serve an OpenAI-compatible API. The devil is in the details.

Ollama: The Developer's Best Friend

Ollama is Docker for AI models. One command to pull, one command to run.

bash

Install

curl -fsSL https://ollama.ai/install.shsh

Pull and run a model

ollama run llama3.1:8b

That's it. You're chatting with a local LLM.

What Ollama Does Well

Model management is effortless:

bash

List available models

ollama list

Pull specific quantizations

ollama pull llama3.1:70b-instruct-q4_K_M

ollama pull codellama:13b

ollama pull mistral:7b

Remove models

ollama rm llama3.1:8b

Create custom models

cat << 'EOF' > Modelfile

FROM llama3.1:8b

SYSTEM "You are a senior Python developer. Be concise."

PARAMETER temperature 0.3

PARAMETER num_ctx 8192

EOF

ollama create python-dev -f Modelfile

ollama run python-dev

The API is simple:

python

import requests

response = requests.post("http://localhost:11434/api/chat", json={
"model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": False

})

print(response.json()["message"]["content"])

Or use the OpenAI-compatible endpoint:

python

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
response = client.chat.completions.create(
model="llama3.1:8b",
    messages=[{"role": "user", "content": "Hello!"}]

)

Ollama's Limitations

Single-user optimized — doesn't handle concurrent requests well

No continuous batching — processes one request fully before starting the next

No tensor parallelism — can't split a model across multiple GPUs efficiently

Limited monitoring — no built-in metrics, no Prometheus integration

Model library is curated — can't run arbitrary HuggingFace models without conversion

Ollama is perfect for: development, prototyping, personal use, small teams. It's NOT for serving 100 concurrent users.

🔧

vLLM: The Production Powerhouse

vLLM (Virtual Large Language Model) was built for one thing: maximum throughput on GPUs.

bash

Install

pip install vllm

Serve a model

vllm serve meta-llama/Llama-3.1-8B-Instruct \

--host 0.0.0.0 \

--port 8000

vLLM's Secret Weapon: PagedAttention

The #1 innovation in vLLM is PagedAttention — a memory management technique for the KV cache (see Episode 5 and Episode 12).

Traditional inference engines allocate KV cache memory as contiguous blocks:

Traditional:

Request 1: [████████░░░░░░░░] (allocated 16 blocks, using 8)

Request 2: [██████░░░░░░░░░░] (allocated 16 blocks, using 6)

Wasted: ░░░░░░░░░░░░░░░░ (18 blocks wasted!)

vLLM uses paged memory (like an OS manages virtual memory):

PagedAttention:

Request 1: [██][██][██][██] (4 pages, exactly what's needed)

Request 2: [██][██][██] (3 pages, exactly what's needed)

Free pool: [░░][░░][░░][░░][░░][░░][░░][░░][░░] (available for new requests)

Result: 2-4x more concurrent requests in the same VRAM. This is the single biggest reason vLLM wins in production.

Continuous Batching

Ollama processes requests sequentially. vLLM uses continuous batching:

Sequential (Ollama):

Request A: [████████████████] done → start B

Request B: [████████████] done → start C

Continuous batching (vLLM):

Request A: [████████████████] done

Request B: [████████████████] done

Request C: [████████████████] done

↑ New requests join the batch without waiting

When Request A finishes, its slot is immediately filled by a new request. The GPU never idles waiting for all requests in a batch to complete.

vLLM in Production

bash

Multi-GPU (tensor parallelism)

vllm serve meta-llama/Llama-3.1-70B-Instruct \

--tensor-parallel-size 4 \

--host 0.0.0.0 \

--port 8000

With quantization

vllm serve meta-llama/Llama-3.1-70B-Instruct \

--quantization awq \

--host 0.0.0.0

Docker deployment

docker run --gpus all \

-p 8000:8000 \

vllm/vllm-openai:latest \

--model meta-llama/Llama-3.1-8B-Instruct

Production features:

Prometheus metrics — vllm:num_requests_running, vllm:gpu_cache_usage_perc

Tensor parallelism — split one model across multiple GPUs

Pipeline parallelism — chain stages across GPUs

Speculative decoding — draft-verify for faster generation (Episode 14)

LoRA hot-swapping — serve multiple fine-tuned adapters from one base model

Prefix caching — shared system prompts cached across requests

vLLM's Limitations

GPU required — no CPU inference mode

NVIDIA-focused — AMD ROCm support exists but lags

Higher startup time — model loading takes longer than Ollama

More complex configuration — many knobs to tune for optimal performance

No built-in model management — you manage downloads yourself

📊

TGI (Text Generation Inference): Hugging Face's Engine

TGI is Hugging Face's production inference server. If your models live on the Hugging Face Hub, TGI is the most natural choice.

bash

Docker (recommended)

docker run --gpus all \

-p 8080:80 \

ghcr.io/huggingface/text-generation-inference:latest \

--model-id meta-llama/Llama-3.1-8B-Instruct

Or install from source

cargo install text-generation-router

What Makes TGI Different

Rust-based router: TGI's HTTP server and request scheduler are written in Rust — fast, memory-safe, and efficient at handling concurrent connections.

Tight HF Hub integration:

bash

Directly serve any HF model

docker run --gpus all -p 8080:80 \

ghcr.io/huggingface/text-generation-inference:latest \

--model-id mistralai/Mistral-7B-Instruct-v0.3

With quantization

docker run --gpus all -p 8080:80 \

ghcr.io/huggingface/text-generation-inference:latest \

--model-id meta-llama/Llama-3.1-70B-Instruct \

--quantize bitsandbytes-nf4

Flash Attention and Paged Attention: TGI implements both, giving it competitive throughput with vLLM.

Token streaming out of the box:

python

import requests

response = requests.post(
"http://localhost:8080/generate_stream",
    json={
        "inputs": "Write a haiku about coding:",
        "parameters": {"max_new_tokens": 50}
    },
    stream=True

)

for line in response.iter_lines():

if line:
        print(line.decode(), end="", flush=True)

TGI's Limitations

Docker-heavy — while installable from source, Docker is the primary deployment method

Less flexible than vLLM — fewer quantization options, less customizable

Slower adoption of latest optimizations — vLLM tends to ship new features faster

GPU required — no CPU inference

Head-to-Head Benchmarks

Real numbers matter more than marketing. Here's what you'll see on an A100 80GB:

Single-User Latency (Llama 3.1 8B)

Engine Time to First Token Generation Speed

Multi-User Throughput (Llama 3.1 8B, 32 concurrent users)

Engine Total Throughput Avg Latency per User

The gap is dramatic. With 32 concurrent users, Ollama is 43x slower than vLLM because it handles one request at a time. vLLM's continuous batching and PagedAttention make the difference.

Memory Efficiency (Llama 3.1 70B, A100 80GB)

Engine Max Concurrent Requests VRAM Usage at Max

vLLM squeezes more concurrent requests out of the same GPU thanks to PagedAttention's efficient memory management.

Decision Matrix: Which One Should You Use?

Are you building a product that serves multiple users?

├── Yes → How many concurrent users?

│ ├── < 5 → TGI or vLLM (either works)

│ ├── 5-50 → vLLM (PagedAttention wins)

│ └── 50+ → vLLM with tensor parallelism

└── No → Is it for personal/local use?

├── Yes → Ollama (easiest setup)
    └── No → Development/testing?
        ├── Yes → Ollama (fast iteration)
        └── No → vLLM (production-ready from day 1)

Quick Reference

Scenario Best Choice Why

🔬

Migration Path

Most teams follow this trajectory:

Day 1: Ollama locally → prove the concept works

Week 1: Ollama on a GPU server → serve the team

Month 1: vLLM on GPU → serve production users

Month 6: vLLM + multiple GPUs → scale

The good news: all three serve an OpenAI-compatible API. Your application code doesn't change when you swap engines. Change the base URL, and everything works.

python

This code works with ALL THREE engines:

from openai import OpenAI

client = OpenAI(
base_url="http://localhost:8000/v1",  # Change port per engine
    api_key="not-needed"

)

response = client.chat.completions.create(
model="llama3.1:8b",
    messages=[{"role": "user", "content": "Hello!"}]

)

🛡️

Practical Takeaways

Ollama for local dev, vLLM for production — this is the standard path

PagedAttention is why vLLM wins at scale — 2-4x more concurrent users per GPU

Continuous batching prevents GPU idle time — critical for multi-user workloads

All three speak OpenAI API — your application code stays the same

TGI is solid but vLLM is eating its lunch — vLLM ships optimizations faster

Don't over-engineer early — start with Ollama, graduate to vLLM when you need scale

📦

What's Next?

Episode 12: KV Cache — We've mentioned it in every episode since Episode 5. Time to go deep. What exactly is stored in the KV cache? Why does it eat so much memory? And what clever tricks (sliding window, multi-query attention, grouped-query attention) keep it under control?

← Previous

Ep 10: llama.cpp

Next →

Ep 12: KV Cache

Next: Episode 12 — KV Cache

Open ChatGPT. Type a message. Get a response in 2 seconds. Now continue the conversation for 30 messages. Notice something?

This is part of a 98-episode series covering AI engineering from tokens to production deployment.