Quantization: Running 70B Models on Your Laptop

Llama 3.1 70B has 70 billion parameters. In FP16, that's 140 GB. Your GPU has 24 GB of VRAM. Math says it doesn't fit.

But people run it. On a single RTX 4090. On a MacBook Pro. On a laptop with 16 GB RAM.

The trick: quantization. Instead of storing each parameter as a 16-bit number, compress it to 4 bits. 140 GB becomes 35 GB. Now it fits.

But you just threw away 75% of the information. Doesn't that destroy the model?

Surprisingly... not really.

What Quantization Actually Does

Every parameter in a neural network is a decimal number like 0.00347 or -1.2891. In full precision (FP32), each number takes 32 bits — 4 bytes. In half precision (FP16), 16 bits — 2 bytes.

Quantization reduces the number of bits per parameter:

FP32: 0.00347000 → 32 bits → 4 bytes per parameter

FP16:  0.00347    → 16 bits → 2 bytes per parameter
INT8:  3          →  8 bits → 1 byte per parameter
INT4:  1          →  4 bits → 0.5 bytes per parameter

Wait — how does 0.00347 become 3 or 1? Through mapping.

The Mapping

Take a range of values and map them to a smaller set of integers:

FP16 range: [-1.5, -0.8, -0.2, 0.1, 0.4, 0.9, 1.3]

INT4 mapping (16 possible values: 0 to 15):

-1.5 → 0

-0.8 → 3

-0.2 → 6

0.1 → 7

0.4 → 9

0.9 → 12

1.3 → 15

Scale factor: (1.3 - (-1.5)) / 15 = 0.187

Zero point: 8

To reconstruct: value ≈ (quantized_int - zero_point) × scale

0 → (0 - 8) × 0.187 = -1.496  (original: -1.5, error: 0.004)
7 → (7 - 8) × 0.187 = -0.187  (original:  0.1, error: 0.287!)

Notice: the reconstruction isn't perfect. There's quantization error. The question is: how much error can the model tolerate before quality degrades noticeably?

The Model Size Math

This is the formula everyone in AI infra uses daily:

Model size (GB) = parameters × bits_per_weight / 8 / 1,000,000,000

Llama 3.1 70B:

FP32: 70B × 4 bytes = 280 GB

FP16: 70B × 2 bytes = 140 GB

Q8:    70B × 1 byte   =  70 GB

Q6_K: 70B × 0.75 = 52.5 GB

Q4_K_M:70B × 0.5 = 35 GB

Q2_K: 70B × 0.25 = 17.5 GB

Q4_K_M is the sweet spot for most people. 4.8 bits per weight. You lose some quality, but the model still works remarkably well.

🔧

Quality vs Speed Tradeoff: Perplexity

How do we measure quality loss from quantization? Perplexity — a standard metric that measures how "surprised" the model is by real text. Lower perplexity = better predictions = higher quality.

Real benchmarks for Llama 3.1 8B:

Quantization Size Perplexity Speed (tok/s) Quality

FP16 16 GB 6.14 (baseline) 25 Perfect

The pattern: Q8 to Q4 loses almost nothing. Q3 starts to feel different. Q2 is where quality falls off a cliff — the model starts making more mistakes, loses nuance, and occasionally produces gibberish.

The Sweet Spots

Q8: When you want near-perfect quality and have the VRAM

Q5_K_M / Q4_K_M: Best bang for buck — barely noticeable quality loss

Q3_K_M: Acceptable for chat, not for critical tasks

Q2_K: Emergency mode only — when nothing else fits

📊

The Format Zoo: GGUF, GPTQ, AWQ

GGUF (GPT-Generated Unified Format)

Created by the llama.cpp project (covered in Episode 10). GGUF is the standard for CPU and hybrid CPU/GPU inference.

bash

GGUF files on HuggingFace

Example: TheBloke/Llama-2-70B-GGUF

bartowski/Meta-Llama-3.1-70B-Instruct-GGUF/

├── Meta-Llama-3.1-70B-Instruct-Q2_K.gguf (17.5 GB)

├── Meta-Llama-3.1-70B-Instruct-Q3_K_M.gguf (28 GB)

├── Meta-Llama-3.1-70B-Instruct-Q4_K_M.gguf (35 GB)

├── Meta-Llama-3.1-70B-Instruct-Q5_K_M.gguf (45 GB)

├── Meta-Llama-3.1-70B-Instruct-Q6_K.gguf (52 GB)

└── Meta-Llama-3.1-70B-Instruct-Q8_0.gguf (70 GB)

Why GGUF?

Works on CPU, GPU, or mixed (some layers on GPU, rest on CPU)

Self-contained: model weights + tokenizer + metadata in one file

Supported by llama.cpp, Ollama, LM Studio, GPT4All

The K-quants (Q4_K_M, Q5_K_M, etc.) use mixed precision — important layers get more bits, less important layers get fewer

GPTQ (GPT-Quantized)

A GPU-only quantization method. Uses a calibration dataset to find the optimal way to quantize each layer.

python

Using GPTQ with transformers

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
"TheBloke/Llama-2-70B-GPTQ",
    device_map="auto"

)

Why GPTQ?

Higher quality than naive quantization (calibration-based)

Faster on GPU than GGUF

Requires GPU — no CPU fallback

Supported by vLLM, text-generation-inference, ExLlamaV2

AWQ (Activation-Aware Weight Quantization)

The newest of the three. Like GPTQ, it's GPU-focused and uses calibration. But AWQ is smarter about which weights matter most.

The key insight: not all weights are equally important. Some weights consistently produce large activations — these "salient" weights need more precision. AWQ protects them.

Traditional quantization: All weights get 4 bits equally

AWQ: Important weights get more protection

Less important weights get aggressively quantized

Why AWQ?

Better quality than GPTQ at the same bit width

Faster inference than GPTQ on modern GPUs

Supported by vLLM (default recommendation), TGI

Which Format to Use?

Scenario Format Why

How Quantization Actually Works (Under the Hood)

Round-to-Nearest (RTN)

The simplest approach. Take each weight, map it to the nearest quantized value.

Weight: 0.347

Quantization grid: [0.0, 0.25, 0.5, 0.75, 1.0]

Nearest: 0.25

Error: 0.347 - 0.25 = 0.097

RTN is fast but introduces significant error. It's what you get with basic GGUF Q4_0 quantization.

K-Quant (GGUF's Smart Method)

GGUF K-quants (Q4_K_M, Q5_K_S, etc.) use mixed precision across layers:

Attention layers: Higher precision (Q6 or Q5)

Feed-forward layers: Lower precision (Q4 or Q3)

Embedding layers: Full precision (Q8 or FP16)

The logic: attention layers are more sensitive to quantization error. Feed-forward layers are more robust. By allocating bits where they matter most, K-quants achieve better quality at the same average bit rate.

That's what the K and M/S/L mean:

K: K-quant method (mixed precision)

S: Small (more compression, lower quality)

M: Medium (balanced)

L: Large (less compression, higher quality)

GPTQ / AWQ: Calibration-Based

These methods run a small dataset through the model and measure which weights matter most:

python

Simplified GPTQ process:

1. Run 128 text samples through the model

2. For each layer, measure the Hessian (sensitivity of output to weight changes)

3. Quantize weights in order of importance

4. After quantizing each weight, adjust remaining weights to compensate

This "adjust remaining weights" step is the magic — it distributes the quantization error across the layer instead of letting it accumulate.

Try It Yourself — Quantize and Compare

bash

Using Ollama (easiest)

ollama pull llama3.1:8b # Default Q4_K_M

ollama pull llama3.1:8b-text-q8_0 # Q8 version

Compare quality:

echo "Explain quantum entanglement"ollama run llama3.1:8b echo "Explain quantum entanglement"ollama run llama3.1:8b-text-q8_0

Check actual file sizes:

ls -lh ~/.ollama/models/blobs/

Or using llama.cpp directly:

bash

Quantize a model yourself

./llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

Measure perplexity

./llama-perplexity -m model-f16.gguf -f wiki.test.raw

./llama-perplexity -m model-q4_k_m.gguf -f wiki.test.raw

🔬

Practical Takeaways

Quantization trades precision for size — 4-bit models are 4x smaller than FP16 with minimal quality loss

Q4_K_M is the sweet spot — best balance of size, speed, and quality for most use cases

Below Q3, quality degrades fast — don't go to Q2 unless absolutely necessary

GGUF for local, AWQ for production GPUs — match the format to your hardware

K-quants are smarter — they give important layers more bits, so Q4_K_M beats Q4_0

Perplexity is your quality metric — track it when comparing quantization levels

🛡️

What's Next?

Episode 10: llama.cpp — The open-source project that made local AI possible. How one developer's C++ code lets you run billion-parameter models on a laptop with no Python, no PyTorch, no CUDA SDK. mmap, GGUF, Metal, and the magic of efficient inference.

← Previous

Ep 8: CPU vs GPU Inference

Next →

Ep 10: llama.cpp

Next: Episode 10 — llama.cpp

In March 2023, a Bulgarian developer made Meta's LLaMA run on a MacBook. That single project changed AI forever.