VRAM Math: How to Calculate If a Model Fits

"Can I run Llama 3 70B on my GPU?"

Every AI engineer asks this question. And most just Google it and hope someone posted a benchmark. But what if you could calculate it yourself — in your head, in 30 seconds?

That's what this episode gives you. A practical formula for VRAM requirements that works for any model, any precision, any batch size. Once you learn this, you'll never wonder "will it fit?" again. You'll know.

The Formula

Here's the master formula:

Total VRAM = Model Weights + KV Cache + Activations + Overhead

Where:

Model Weights = Parameters × Bytes per Parameter

KV Cache = 2 × Layers × Hidden_dim × Seq_len × Batch_size × Bytes

Activations   = ~10-20% of model weights (during inference)
Overhead      = CUDA context + framework buffers (~500MB-1GB)

Let's break each piece down.

Part 1: Model Weights

This is the big one — and the simplest to calculate.

Bytes per Parameter

Precision Bytes per Param Formula

FP32 4 bytes params × 4

FP16 / BF16 2 bytes params × 2

FP8 1 byte params × 1

INT8 1 byte params × 1

INT4 (Q4) 0.5 bytes params × 0.5

GPTQ/AWQ 4-bit ~0.55 bytes params × 0.55 (quantization overhead)

Quick Calculations

Llama 3 8B:

FP16: 8B × 2 bytes = 16 GB

INT8: 8B × 1 byte = 8 GB

INT4: 8B × 0.5 = 4 GB

Llama 3 70B:

FP16: 70B × 2 bytes = 140 GB

INT8: 70B × 1 byte = 70 GB

INT4: 70B × 0.5 = 35 GB

Llama 3 405B:

FP16: 405B × 2 = 810 GB

INT4: 405B × 0.5 = 202.5 GB

The mental shortcut: For FP16, the VRAM in GB is roughly 2× the parameter count in billions. For INT4, it's roughly half the parameter count.

FP16: VRAM (GB) ≈ Params (B) × 2

INT4: VRAM (GB) ≈ Params (B) × 0.5

INT8: VRAM (GB) ≈ Params (B) × 1

Does It Fit?

🔧

Part 2: KV Cache Memory

The KV Cache stores the key and value vectors for every token in the context window, for every layer, for every attention head. During generation, this cache grows with each token and can consume more VRAM than the model weights for long contexts.

The KV Cache Formula

KV Cache (bytes) = 2 × num_layers × hidden_dim × seq_length × batch_size × bytes_per_value

The "2" is for K (keys) and V (values) — both are cached.

Worked Example: Llama 3 70B

Architecture:

Layers: 80

Hidden dim: 8192

Attention heads: 64

KV heads: 8 (GQA — Grouped Query Attention)

Head dim: 128

With GQA, the effective KV hidden dim = num_kv_heads × head_dim = 8 × 128 = 1024

KV Cache for 1 sequence of 4K tokens (FP16):

= 2 × 80 layers × 1024 × 4096 tokens × 2 bytes

= 2 × 80 × 1024 × 4096 × 2

= 1,342,177,280 bytes

≈ 1.25 GB

KV Cache for 1 sequence of 128K tokens (FP16):

= 2 × 80 × 1024 × 131072 × 2

≈ 40 GB ← Half the VRAM just for one sequence's KV cache!

KV Cache Scaling

Key insight: GQA (Grouped Query Attention) saves enormous KV cache memory. Without GQA, Llama 3 70B would use 8x more KV cache (64 KV heads instead of 8). That's why GQA is standard in modern architectures.

Batch Size Impact

Each request in a batch needs its own KV cache. This multiplies linearly:

Batch size 1:  1 × 1.25 GB = 1.25 GB
Batch size 8:  8 × 1.25 GB = 10 GB
Batch size 32: 32 × 1.25 GB = 40 GB
Batch size 64: 64 × 1.25 GB = 80 GB  ← Exceeds the entire VRAM!

This is why serving AI at high throughput requires careful VRAM budgeting. The model weights are fixed, but KV cache scales with both sequence length AND concurrent users.

📊

Part 3: Activations

Activations are the intermediate values computed during the forward pass — the output of each layer before it feeds into the next.

During Inference

Activations are relatively small during inference because you only need the current layer's output:

Activation memory ≈ batch_size × seq_len × hidden_dim × bytes

For Llama 3 70B, 1 sequence, 4K tokens:

≈ 1 × 4096 × 8192 × 2 bytes ≈ 64 MB (small!)

During Training

Activations during training are massive because you need to store all intermediate values for backpropagation:

Training activation memory ≈ num_layers × batch_size × seq_len × hidden_dim × bytes × factor

The "factor" accounts for multiple intermediate values per layer:

• Attention scores
• QKV projections
• FFN intermediate activations
• Layer norm values

For Llama 3 70B training:

≈ 80 × 1 × 4096 × 8192 × 2 × 10 ≈ 50+ GB per sample

This is why training requires far more VRAM than inference and why techniques like gradient checkpointing (recomputing activations instead of storing them) are essential.

Part 4: Putting It All Together

Inference VRAM Budget

Llama 3 70B, INT4, batch_size=1, 4K context:
Model weights:     70B × 0.5 = 35.0 GB

KV cache: 1.25 GB (FP16, 4K tokens)

Activations: ~0.1 GB

Overhead: ~1.0 GB

─────────────────────────────

Total:             ~37.4 GB    → Fits on A100 80GB ✅
→ Fits on 2× RTX 4090 48GB ✅
Llama 3 70B, FP16, batch_size=16, 8K context:
Model weights:     70B × 2 = 140.0 GB

KV cache: 16 × 2.5 = 40.0 GB

Activations: ~2.0 GB

Overhead: ~1.0 GB

─────────────────────────────

Total:             ~183 GB     → Needs 3× A100 80GB
→ Or 2× H200 141GB

Training VRAM Budget

Training requires far more — you need:

Model weights (in FP32 for master copy)

Optimizer states (Adam stores 2 extra copies: momentum + variance)

Gradients (same size as weights)
Activations (stored for backpropagation)

Training VRAM (rough estimate):

≈ 4 × model_weights_FP32 + activations

For a 7B model:

Weights (FP32): 7B × 4 = 28 GB
Optimizer state: 7B × 8 = 56 GB (Adam: 2 × FP32 copies)
Gradients: 7B × 4 = 28 GB

Activations: ~30 GB

─────────────────────────

Total: ~142 GB → Needs 2× A100 80GB

Rule of thumb for training: You need roughly 18-20 bytes per parameter for full fine-tuning with Adam optimizer. So a 7B model needs ~130-140 GB.

The Cheat Sheet

Print this. Tattoo it. Whatever works:

Quick VRAM Estimates (Inference)

Params INT4 INT8 FP16

Add 10-30% for KV cache and overhead at typical batch sizes.

GPU Memory Quick Reference

GPU VRAM Bandwidth Fits (INT4, batch=1)

RTX 3060 12 GB 360 GB/s Up to ~20B

RTX 3090 24 GB 936 GB/s Up to ~40B

RTX 4090 24 GB 1,008 GB/s Up to ~40B

*Mac uses unified memory (shared CPU/GPU), which means the full amount is available for models.

🔬

Common Mistakes

Mistake 1: Forgetting KV Cache

"70B in INT4 is 35 GB, fits on my A100 80GB!"

Yes, but with batch_size=32 and 8K context:

KV cache = 32 × 2.5 GB = 80 GB ← EXCEEDS TOTAL VRAM

Your model fits but you can't serve any traffic.

Mistake 2: Confusing INT4 with FP16

"I downloaded the Llama 3 70B FP16 model, it should be 35 GB..."
No. FP16 = 2 bytes per param = 140 GB.
INT4 = 0.5 bytes per param = 35 GB.

You downloaded 140 GB. It won't fit.

Mistake 3: Ignoring Quantization Overhead

Real INT4 models (GPTQ, AWQ) store per-group quantization scales and zeros, adding ~10% overhead:

Theoretical INT4: 70B × 0.5 = 35.0 GB

Actual GPTQ-4bit: 70B × 0.55 ≈ 38.5 GB

🛡️

Practical Takeaways

FP16 VRAM ≈ params × 2 — the simplest formula in AI engineering

KV cache can exceed model weights — especially with long contexts and large batches

GQA saves KV cache memory — 8 KV heads vs 64 can mean 8x less cache

Training needs ~18-20 bytes per parameter — 10x more than inference

Always add 10-30% for overhead — CUDA context, framework buffers, fragmentation

Batch size × sequence length determines serving cost — not just model size

📦

What's Next?

Episode 66: Multi-GPU Setups — When one GPU isn't enough, how do you split a model across multiple GPUs? Tensor parallelism, pipeline parallelism, data parallelism — and when you need each one.

← Previous

Ep 64: GPU Architecture Basics

Next →

Ep 66: Multi-GPU Setups

Next: Episode 66 — Multi-GPU Setups

Llama 3 405B in FP16 is 810 GB. The biggest GPU has 192 GB. You need at least 5 GPUs. Here's how model parallelism works.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.