CPU vs GPU Inference: Why Your Graphics Card Matters More Than Your Processor

A friend bought an RTX 4090 for ₹1,60,000. His wife thought it was for gaming. It wasn't. He runs Llama 3.1 70B on it.

Another friend tried running the same model on his ₹80,000 laptop with a Ryzen 9 and 32GB RAM. No GPU. It works — at 0.8 tokens per second. That's roughly one word every 1.2 seconds. Reading the response is faster than waiting for it.

Why is a "graphics card" 50x faster at AI than a top-tier processor? And when is CPU actually fine?

The Fundamental Difference: Few Smart Workers vs. Many Simple Workers

A CPU is like having 8 extremely skilled engineers. Each one can handle complex logic, branching decisions, and varied tasks. Brilliant generalists.

A GPU is like having 16,000 factory workers. Each one can only do one simple math operation. But they all work simultaneously.

CPU (M2 Pro) GPU (RTX 4090)

That last row — memory bandwidth — is the real story. Not cores. Not clock speed. Memory bandwidth.

Why AI Needs Parallel Math

Matrix multiplication. That's 90%+ of what happens during inference.

From Episode 3, you know a model is billions of numbers (parameters) organized in matrices. Running inference means multiplying your input vector by these matrices, layer by layer.

A single matrix multiplication for a 70B model:

Input: [1 × 8192] vector

Weight matrix: [8192 × 8192]

Output: [1 × 8192] vector

Operations needed: 8192 × 8192 = 67 million multiply-adds

And that's ONE layer. A 70B model has 80 layers, each with multiple matrix multiplications. Every single token requires billions of arithmetic operations.

A CPU does these billions of operations mostly sequentially — a few at a time across 8-12 cores. A GPU does them in massive parallel batches across 16,000+ cores. Each core is dumber, but there are thousands of them hitting the same problem simultaneously.

CPU approach: [core1: mul] [core2: mul] ... [core12: mul]

Then next batch... then next... (millions of batches)

GPU approach: [core1: mul] [core2: mul] ... [core16384: mul]

Then next batch... (thousands of batches)

🔧

CUDA Cores: What They Actually Are

CUDA (Compute Unified Device Architecture) is NVIDIA's parallel computing platform. CUDA cores are the individual processing units on an NVIDIA GPU.

But not all CUDA cores are equal. Modern GPUs also have Tensor Cores — specialized circuits designed specifically for matrix multiplication:

Core Type What It Does Speed

CUDA Core General parallel math (add, multiply) 1x

Tensor Core Matrix multiply-accumulate (MMA) 8-16x for AI workloads

An RTX 4090 has 16,384 CUDA cores AND 512 Tensor Cores. For AI inference, the Tensor Cores do most of the heavy lifting.

The GPU Lineup for AI

GPU VRAM Bandwidth FP16 TFLOPS Price

RTX 3060 12GB 360 GB/s 25.3 ₹25,000

RTX 4060 Ti 16GB 288 GB/s 22.1 ₹35,000

RTX 4090 24GB 1,008 GB/s 82.6 ₹1,60,000

For local inference, the RTX 4090 is the sweet spot. For production, it's A100 or H100.

📊

The Real Bottleneck: Memory Bandwidth

Here's the counterintuitive truth: inference is usually limited by memory speed, not compute speed.

Why? The model weights need to be read from memory for every single token generated. A 70B model in FP16 is ~140 GB. For each token, the GPU needs to read through those 140 GB of weights.

Token generation time ≈ model_size / memory_bandwidth

RTX 4090: 140 GB / 1,008 GB/s = 139 ms per token ≈ 7 tok/s

A100:      140 GB / 2,039 GB/s =  69 ms per token ≈ 14 tok/s
H100:      140 GB / 3,350 GB/s =  42 ms per token ≈ 24 tok/s
CPU (DDR5): 140 GB / 200 GB/s = 700 ms per token ≈ 1.4 tok/s (M2 Pro)

That's why the CPU is 10x slower — not because of compute, but because system RAM is 10x slower than VRAM. The arithmetic is fast enough on both. Getting the data to the arithmetic units is the bottleneck.

This is called being memory-bandwidth bound — and it's the default state for single-user inference.

VRAM: The Hard Limit

VRAM (Video RAM) is the GPU's dedicated memory. Unlike system RAM, it's right next to the GPU cores — that's why it's so fast.

But it's limited. And everything needs to fit:

VRAM usage = model weights + KV cache + activations + overhead

For Llama 3.1 70B on 24GB GPU:

Model weights (FP16): 140 GB  ← DOESN'T FIT.

Model weights (Q4): 35 GB ← Still doesn't fit in 24GB.

Model weights (Q4):    35 GB across 2× RTX 4090 = 17.5 GB each ✓

This is why quantization matters (next episode). If you can't fit the model in VRAM, you have two choices:

Quantize — reduce precision, shrink the model

Offload to CPU — spill layers to system RAM (much slower)

Use multiple GPUs — split the model across devices

Option 2 is why CPU inference "works" but is painfully slow. The model lives in system RAM, and system RAM is 10x slower than VRAM.

When CPU Is Actually Fine

GPU isn't always necessary. Here's the honest breakdown:

Model Size CPU OK? GPU Recommended? Why

CPU sweet spots:

Tiny models for edge devices — running a 1B model on a Raspberry Pi for local text classification

Batch processing where latency doesn't matter — process 10,000 documents overnight on a 48-core server

Apple Silicon Macs — M1/M2/M3/M4 have unified memory with decent bandwidth (~68 GB/s for base M1 up to 800+ GB/s for M4 Ultra), blurring the CPU/GPU line

Quick experimentation — just want to test a prompt, don't care about speed

The Apple Silicon Exception

Apple's M-series chips deserve special mention. They have unified memory — the CPU and GPU share the same RAM pool. An M2 Max with 96GB unified memory can load a 70B quantized model entirely in "GPU-accessible" memory.

M2 Max (96GB):

Memory bandwidth: ~400 GB/s

Llama 3.1 70B Q4: ~35 GB → fits in memory

Inference speed: ~10-15 tok/s

RTX 4090 (24GB):

Can't even fit the 35GB model without offloading!

For local inference, high-memory Apple Silicon is surprisingly competitive. Not as fast per-token as a datacenter GPU, but it can handle models that don't fit on consumer NVIDIA cards.

🔬

Putting It All Together: Decision Matrix

Do you need real-time responses (< 100ms)?

├── Yes → GPU required (or high-end Apple Silicon)

└── No → How big is the model?

├── < 7B → CPU is fine, GPU is faster
    ├── 7-30B → GPU strongly recommended
    └── 30B+ → GPU required, possibly multiple

Cost Comparison: Cloud GPU vs CPU

Instance Specs Hourly Cost 70B Q4 Speed

AWS c7g.4xlarge (CPU) 16 vCPU, 32GB $0.58/hr ~1 tok/s

AWS g5.xlarge (GPU) A10G 24GB $1.01/hr ~15 tok/s

AWS p4d.24xlarge (GPU) 8× A100 80GB $32.77/hr ~100 tok/s

The GPU instance is 15x faster at only 1.7x the cost. Per-token, GPU is almost 10x cheaper than CPU.

🛡️

Try It Yourself — Measure Your Hardware

bash

Install llama.cpp (covered in Episode 10)

Then benchmark your hardware:

CPU-only inference

./llama-bench -m model.gguf -t 8 -ngl 0

GPU inference (offload all layers)

./llama-bench -m model.gguf -ngl 99

Partial GPU offload (if model doesn't fit in VRAM)

./llama-bench -m model.gguf -ngl 20 # 20 layers on GPU

The -ngl flag (number of GPU layers) is your main tuning knob. More layers on GPU = faster, until you run out of VRAM.

📦

Practical Takeaways

GPUs are faster because of memory bandwidth, not just more cores — getting data to the compute units is the bottleneck

VRAM is the hard constraint — if the model doesn't fit, you're offloading to slow system RAM

Tensor Cores do the real AI work — regular CUDA cores are secondary for inference

CPU is fine for small models (< 7B) — don't overthink hardware for quick experiments

Apple Silicon blurs the line — unified memory with decent bandwidth makes Macs surprisingly good for local AI

Per-token, GPU is much cheaper than CPU — even though GPU instances cost more per hour

🚀

What's Next?

Episode 9: Quantization — We keep saying "70B model = 140 GB." But people run it on 24 GB cards. How? By squeezing each parameter from 16 bits down to 4 bits. What does that trade away? When does quality break? GGUF, GPTQ, AWQ — the alphabet soup decoded.

← Previous

Ep 7: Why LLMs Hallucinate

Next →

Ep 9: Quantization

Next: Episode 9 — Quantization

Llama 3.1 70B needs 140 GB in FP16. Your GPU has 24 GB. Math says it doesn't fit. Quantization says hold my beer.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.