GPU Architecture Basics: Why GPUs Are AI's Engine

Your CPU has 8 cores. Maybe 16 if you're fancy. A single NVIDIA H100 GPU has 16,896 CUDA cores and 528 Tensor Cores.

But it's not just about the number of cores. CPUs and GPUs are fundamentally different machines, designed for fundamentally different problems. A CPU is a Swiss Army knife — versatile, precise, handles anything. A GPU is a combine harvester — does one thing (parallel math) and does it at a scale that makes CPUs look like hand tools.

Understanding GPU architecture isn't just academic. It explains why quantization works, why batching matters, why tensor parallelism exists, and why an H100 costs $30,000. Every optimization we've covered in this series traces back to the hardware.

CPU vs GPU: The Fundamental Difference

CPU: Few Cores, Very Smart

A CPU core is a general-purpose computing marvel. It handles:

Branch prediction (guessing which if statement path to take)

Out-of-order execution (reordering instructions for speed)

Complex cache hierarchies (L1, L2, L3)

Virtual memory management

Interrupt handling

Each core is like a PhD mathematician — can solve any problem, handles complex logic, works independently.

GPU: Thousands of Cores, Simple Each

A GPU core (CUDA core) does one thing: multiply and add numbers. That's it. No branch prediction. Minimal cache. Limited control flow. But there are thousands of them, all working simultaneously.

Each core is like a factory worker doing one repetitive task — simple but incredibly fast when thousands work in parallel.

Task: Multiply two 4096×4096 matrices

CPU (16 cores):

Each core handles a section of the matrix

Serial inner loops, complex scheduling

Time: ~500ms

GPU (16,896 CUDA cores):

Each core handles a tiny piece

Massively parallel, simple math

Time: ~2ms

GPU is 250x faster at this specific task.

Why AI Needs GPUs

AI is almost entirely matrix multiplication:

Forward pass: output = input × weights + bias (matrix multiply)

Attention: scores = Q × K^T (matrix multiply)

output = scores × V                 (matrix multiply)

Backward pass: gradients = error × weights^T (matrix multiply)

A single transformer forward pass for a 70B model involves thousands of matrix multiplications. This is the most parallelizable workload in computing — perfect for GPUs.

NVIDIA GPU Architecture: Inside the Chip

Let's break down what's inside an H100:

The Hierarchy

GPU Chip (H100)

└── 132 Streaming Multiprocessors (SMs)

└── Each SM contains:
            ├── 128 CUDA Cores (FP32)
            ├── 4 Tensor Cores (4th gen)
            ├── 64 INT32 cores
            ├── 256 KB Register File
            ├── 256 KB L1 Cache / Shared Memory
            └── Warp schedulers (4 per SM)

Streaming Multiprocessor (SM)

The SM is the fundamental compute unit. Think of it as a mini-processor. The H100 has 132 SMs, each operating semi-independently.

Key concept: warps. The GPU groups 32 threads into a warp. All 32 threads in a warp execute the same instruction at the same time (SIMT — Single Instruction, Multiple Threads). This is why GPUs love uniform computation and hate branching.

Warp (32 threads):

Thread 0: multiply a[0] × b[0]

Thread 1: multiply a[1] × b[1]

Thread 2: multiply a[2] × b[2]

...

Thread 31: multiply a[31] × b[31]

All 32 multiplications happen SIMULTANEOUSLY in one clock cycle.

CUDA Cores vs Tensor Cores

This is the most important distinction for AI:

CUDA Cores — General-purpose floating point units. Each performs one multiply-add per clock cycle:

CUDA Core: a × b + c = result (1 operation per clock)

Tensor Cores — Specialized matrix multiply units. Each performs a 4×4 matrix multiply-accumulate per clock cycle:

Tensor Core: A[4×4] × B[4×4] + C[4×4] = D[4×4] (64 operations per clock!)

That's 64x more operations per clock for matrix math. This is why Tensor Cores dominate AI workloads:

Core Type FP16 Performance (H100) Use Case

CUDA Cores ~130 TFLOPS General compute, non-matrix ops

Tensor Cores ~990 TFLOPS Matrix multiply (attention, FFN)

Tensor Cores are why NVIDIA owns the AI market. They're purpose-built for the exact operation AI needs most.

🔧

Memory Architecture: Where the Real Bottleneck Lives

GPU compute is fast. Getting data to the compute units is the bottleneck. This is the memory wall — and it's the most important concept in AI hardware.

The Memory Hierarchy

Register File: Fastest 256 KB per SM ~20 TB/s bandwidth

↓

L1 Cache/SMEM: Fast 256 KB per SM ~20 TB/s bandwidth

↓

L2 Cache: Medium 50 MB (shared) ~12 TB/s bandwidth

↓

HBM3 (VRAM):       Slow       80 GB             3.35 TB/s bandwidth

↓

PCIe/NVLink: Slowest Host ↔ GPU 64-900 GB/s

HBM: High Bandwidth Memory

GPU VRAM isn't regular RAM. It's HBM (High Bandwidth Memory) — stacked memory chips physically mounted on top of the GPU die:

Regular DDR5 RAM: ~50 GB/s bandwidth

HBM3 (H100): 3,350 GB/s bandwidth (3.35 TB/s)

HBM3e (H200): 4,800 GB/s bandwidth (4.8 TB/s)

67x more bandwidth than regular RAM. This is what makes GPUs fast for AI — not just compute, but the ability to feed data to the compute units fast enough.

Why Memory Bandwidth Matters

During inference, the model weights must be read from VRAM for every forward pass:

Model: Llama 3 70B in FP16 = 140 GB of weights

Each token generation:

• Read 140 GB of weights from VRAM
• Do matrix multiplies (fast)
• Write output (small)

At 3.35 TB/s bandwidth (H100):

Time to read weights = 140 GB / 3,350 GB/s = 0.042 seconds

≈ 24 tokens/second (memory-bound!)

The Tensor Cores could compute MUCH faster,

but they're waiting for data from memory.

This is why inference for large models is memory bandwidth-bound, not compute-bound. The Tensor Cores sit idle most of the time, waiting for weights to arrive from HBM.

📊

The GPU Generations That Matter

GPU Year VRAM Bandwidth FP16 Tensor Key Feature

The generational jumps are massive — each generation roughly doubles both memory bandwidth and compute. This is why a 2-year-old GPU feels ancient in AI.

Precision Formats: Why FP8 and FP4 Exist

GPUs support multiple number formats, and newer GPUs add lower-precision formats specifically for AI:

Format Bits Range Tensor Core Support Use Case

FP32 32 ±3.4×10³⁸ All GPUs Training (master weights)

TF32 19* ±3.4×10³⁸ A100+ Training (Tensor Core default)

FP16 16 ±65,504 V100+ Mixed precision training

BF16 16 ±3.4×10³⁸ A100+ Training (preferred)

FP8 8 ±448 (E4M3) H100+ Inference, some training

INT8 8 -128 to 127 All Quantized inference

INT4 4 -8 to 7 H100+ Quantized inference

FP4 4 — B200 Heavily quantized inference

*TF32 uses 19 bits total but has FP32 range with FP16 mantissa precision.

Now you understand why quantization (Episode 12) works — the hardware literally supports running models at lower precision. FP16 on Tensor Cores gives you 2x the throughput of FP32. FP8 doubles again. INT4 doubles again. Each lower precision means more operations per second and less memory for weights.

70B model memory:

FP32: 280 GB (doesn't fit on any single GPU)

FP16: 140 GB (fits on H200)

FP8: 70 GB (fits on A100/H100 80GB)

INT4: 35 GB (fits on consumer RTX 4090 24GB — close)

NVLink and GPU Interconnects

When one GPU isn't enough, you need to connect multiple GPUs. The interconnect speed determines how well multi-GPU setups work:

Interconnect Bandwidth Use Case

NVLink is 14x faster than PCIe. This is why NVIDIA's DGX systems (with NVLink) vastly outperform DIY multi-GPU setups (with PCIe).

Tensor Parallelism (splitting model across GPUs):

With NVLink: 900 GB/s → minimal overhead → scales well
With PCIe:    64 GB/s → massive overhead → doesn't scale

This is why you can't just buy 8 consumer GPUs and get the same performance as a DGX — the interconnect bottleneck kills performance.

🔬

The CUDA Software Stack

NVIDIA's dominance isn't just hardware — it's the software ecosystem:

Your AI code (PyTorch)

↓

cuDNN (deep learning primitives)
↓
cuBLAS (matrix operations)
↓

CUDA Runtime

↓

CUDA Driver

↓

GPU Hardware

cuBLAS — Optimized matrix multiply library. Years of hand-tuned kernels for every GPU generation.

cuDNN — Deep learning primitives (convolution, attention, normalization). Used by PyTorch and TensorFlow internally.

CUDA — The programming model. Write code that runs on thousands of GPU threads.

This software stack represents 15+ years of investment. AMD (ROCm) and Intel (OneAPI) are catching up, but NVIDIA's software moat is deep.

🛡️

Practical Takeaways

GPUs beat CPUs at AI because of massive parallelism — thousands of cores doing matrix math simultaneously

Tensor Cores are the key innovation — 64 operations per clock for matrix multiply, 8x faster than CUDA cores for AI

Memory bandwidth is usually the bottleneck — inference is limited by how fast you can read weights from VRAM, not compute

Lower precision = more speed — FP16 doubles throughput over FP32, FP8 doubles again — hardware natively supports it

Interconnects matter for multi-GPU — NVLink (900 GB/s) vs PCIe (64 GB/s) is a 14x difference

NVIDIA's moat is software + hardware — CUDA, cuBLAS, cuDNN represent 15 years of optimization

📦

What's Next?

Episode 65: VRAM Math — How do you calculate if a model fits in your GPU? Bytes per parameter, KV cache memory, batch size impact. A practical formula you'll use every day as an AI engineer.

← Previous

Ep 63: What's Next in AI

Next →

Ep 65: VRAM Math

Next: Episode 65 — VRAM Math

Every AI engineer asks: will this model fit on my GPU? Here's how to calculate it yourself — in your head, in 30 seconds.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.