Speculative Decoding: Getting 2-3x Speed for Free

Here's a management trick that works for both humans and AI:

Don't make the senior engineer write every line of code. Have a junior write the first draft. Senior reviews it — approves what's good, fixes what's not. The senior's time is the bottleneck, and reviewing is faster than writing.

Speculative decoding applies this exact principle to LLM inference. A small, fast model drafts multiple tokens. A large, accurate model verifies them in one shot. The result: 2-3x faster generation with mathematically zero quality loss.

Not "negligible" quality loss. Zero. The output is identical to running the large model alone.

The Problem: Sequential Token Generation

From Episode 4, you know LLMs generate one token at a time. Each token depends on the previous one. You can't parallelize this.

Standard generation (70B model):

Step 1: Input → [70B model] → "The"      (50ms)
Step 2: +The  → [70B model] → "quick"    (50ms)
Step 3: +quick → [70B model] → "brown"   (50ms)
Step 4: +brown → [70B model] → "fox"     (50ms)

Total for 4 tokens: 200ms

Each step takes ~50ms for a 70B model (on decent hardware). That's the memory bandwidth bottleneck from Episode 8 — you read 140 GB of model weights for every single token.

But what if you could verify 4 tokens in roughly the same time as generating 1?

The Key Insight: Verification Is Cheaper Than Generation

Here's something non-obvious: checking whether 4 tokens are correct takes about the same time as generating 1 token.

Why? Because the large model can process all 4 draft tokens in parallel (like processing a prompt), but it can only generate new tokens one at a time.

Generate 4 tokens:    4 sequential forward passes → ~200ms
Verify 4 tokens:      1 parallel forward pass     → ~55ms

Generation is sequential (each token depends on the previous). Verification is parallel (all draft tokens are known, just check them all at once — like prefill).

This asymmetry is the foundation of speculative decoding.

🔧

How It Works: Draft and Verify

Step 1: Draft Phase

A small, fast model (the draft model) generates K tokens autoregressively:

Draft model (1B parameters, ~5ms per token):

Input: "The capital of France"

Draft: ["is", "Paris", ",", "and"]

Time: 4 × 5ms = 20ms

Step 2: Verify Phase

The large model (the target model) processes the original input + ALL draft tokens in a single forward pass:

Target model (70B parameters):

Input: "The capital of France" + ["is", "Paris", ",", "and"]

↓       ↓      ↓     ↓

Verify: [✅] [✅] [✅] [❌]

"is"   "Paris"  ","   should be "which"

Time: ~55ms (one forward pass)

Step 3: Accept or Reject

Accept tokens from left to right until the first rejection:

Draft: ["is", "Paris", ",", "and"]

Verified: [ ✅, ✅, ✅, ❌ ]

Accepted: "is", "Paris", "," (3 tokens accepted)

Rejected: "and" → replaced with target model's token: "which"

Final output: "is Paris , which" (4 tokens in ~75ms instead of ~200ms)

The Math

Without speculative decoding:

4 tokens × 50ms each = 200ms

With speculative decoding:

Draft: 4 tokens × 5ms = 20ms
Verify: 1 forward pass = 55ms

Total: 75ms for 4 tokens

Speedup: 200ms / 75ms = 2.67x

And crucially: the accepted tokens are identical to what the target model would have generated. The rejection sampling algorithm guarantees this mathematically.

📊

The Rejection Sampling Guarantee

This is what makes speculative decoding special. It's not an approximation. The acceptance/rejection is based on comparing probability distributions:

For each draft token i:

p(token_i) = target model's probability for this token
q(token_i) = draft model's probability for this token

Accept with probability: min(1, p(token_i) / q(token_i))

If the draft model assigns probability 0.3 to "Paris" and the target model assigns 0.9, acceptance probability = min(1, 0.9/0.3) = 1.0 — always accepted.

If the draft model assigns 0.4 to "and" but the target model assigns 0.05, acceptance probability = min(1, 0.05/0.4) = 0.125 — likely rejected.

When a token is rejected, the algorithm samples a correction from a modified distribution that ensures the overall output distribution matches the target model exactly. Not approximately. Exactly.

This is proven mathematically. The output of speculative decoding is statistically identical to the output of the target model alone.

When It Works (And When It Doesn't)

Acceptance Rate: The Key Metric

The acceptance rate (α) determines the speedup. If the draft model's guesses align well with the target model, most tokens get accepted:

High acceptance rate (α ≈ 0.8):

Draft 8 tokens → ~6.4 accepted → great speedup

Low acceptance rate (α ≈ 0.3):

Draft 8 tokens → ~2.4 accepted → mediocre speedup

(Plus overhead of running the draft model → might be slower!)

What Affects Acceptance Rate?

Factor High Acceptance Low Acceptance

Best case: Code generation with temperature=0. Code is highly predictable — the draft model guesses right 80-90% of the time. Speedup: 2.5-3x.

Worst case: Creative writing with temperature=1.0. Many valid continuations, hard to guess. Speedup: 1.2-1.5x (barely worth the overhead).

The Overhead Budget

Speculative decoding adds cost:

Cost without spec dec: 1 target forward pass per token

Cost with spec dec: K draft forward passes + 1 target forward pass per attempt

If K=8 draft tokens and α=0.8:

Accepted: ~6.4 tokens per attempt

Cost: 8 draft passes (cheap) + 1 target pass (expensive)

Effective cost per accepted token: 1 target pass / 6.4 tokens = 0.16 target passes

Speedup = 1 / 0.16 ≈ 6x in compute per token

Real speedup: ~2.5x (memory bandwidth still applies)

The draft model must be significantly smaller than the target. Running a 7B draft for a 13B target wastes more time on drafting than it saves on verification. A 1-2B draft for a 70B target is the sweet spot.

Draft Model Choices

Option 1: Smaller Model from Same Family

The most common approach. Use Llama 3.1 1B as draft for Llama 3.1 70B:

bash

vLLM with speculative decoding

vllm serve meta-llama/Llama-3.1-70B-Instruct \

--speculative-model meta-llama/Llama-3.1-8B-Instruct \

--num-speculative-tokens 5 \

--speculative-draft-tensor-parallel-size 1

Same tokenizer, similar training data, high acceptance rate.

Option 2: Quantized Self-Draft

Use a heavily quantized version of the SAME model:

Target: Llama 70B FP16 (140 GB, slow)

Draft: Llama 70B Q2 (17 GB, much faster to run)

The draft model has the same architecture and knowledge, just less precise. High acceptance rate because it's literally the same model, blurred.

Option 3: Medusa / Eagle (Self-Speculative)

Instead of a separate draft model, add lightweight "heads" to the target model that predict multiple tokens ahead:

Standard model:     Input → Transformer → [Head] → next token
Medusa model:       Input → Transformer → [Head 0] → token +1
→ [Head 1] → token +2
                                        → [Head 2] → token +3

One forward pass → 3 candidate tokens

Then verify in the next forward pass

No separate draft model needed. The extra heads are small (~1% of model size) and trained cheaply. This is increasingly popular because it avoids the complexity of managing two models.

🔬

Real-World Performance

Benchmarks on A100 80GB:

Setup Model Speed (tok/s) Speedup

By task type:

Task Acceptance Rate Speedup

🛡️

Implementation: Using Speculative Decoding

With vLLM

bash

Basic speculative decoding

vllm serve meta-llama/Llama-3.1-70B-Instruct \

--speculative-model meta-llama/Llama-3.1-8B-Instruct \

--num-speculative-tokens 5

Tune the number of speculative tokens

More tokens = higher potential speedup but lower acceptance rate

3-8 tokens is the typical range

With llama.cpp

bash

Speculative decoding in llama.cpp

./llama-speculative \

-m large-model.gguf \

-md draft-model.gguf \

-ngl 99 \

--draft 8 \

-p "Write a Python function to sort a list"

With Hugging Face Transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer

Load target and draft models

target = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-70B-Instruct")
draft = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-70B-Instruct")
inputs = tokenizer("Write a Python sort function:", return_tensors="pt")

Generate with speculative decoding

outputs = target.generate(
**inputs,
    assistant_model=draft,
    max_new_tokens=200,
    do_sample=False

)

print(tokenizer.decode(outputs[0]))

📦

When NOT to Use Speculative Decoding

It's not always a win:

Already batch-saturated — If you're serving 64 concurrent users, your GPU is busy. Adding a draft model competes for the same resources.

High temperature — Creative tasks with temp > 0.8 have low acceptance rates. The overhead isn't worth it.

Very short outputs — If average output is < 20 tokens, the overhead of loading and running the draft model exceeds the time saved.

VRAM-constrained — The draft model needs to fit alongside the target model. On a 24GB GPU already running a Q4 70B model, there's no room for a draft model.

Latency-sensitive prefill — Speculative decoding speeds up decode (generation), not prefill (prompt processing). If your bottleneck is long prompts, it won't help.

🚀

The Future: Multi-Token Prediction

Meta's research on multi-token prediction takes this further. Instead of training a model to predict 1 token at a time, train it to predict 4 tokens at once:

Standard:   Input → Transformer → 1 token
Multi-token: Input → Transformer → 4 tokens simultaneously

This isn't speculative — the model is genuinely trained to output multiple tokens. Early results show 3x speedup on code generation with minimal quality loss. It's likely coming to major models in 2025-2026.

🔄

Practical Takeaways

Speculative decoding gives 2-3x speed with zero quality loss — mathematically proven identical output

Draft-verify: small model guesses, large model checks — verification is parallel, generation is sequential

Acceptance rate determines speedup — predictable tasks (code, translation) benefit most

Use a draft model 5-10x smaller than the target — 1B draft for 70B target is ideal

Don't use it when GPU is already busy — speculative decoding helps single-user latency, not multi-user throughput

Temperature matters hugely — temp=0 gets 3x speedup, temp=1.0 barely helps

🏗️

What's Next?

Episode 15: Embeddings — We leave the world of text generation and enter the world of meaning. How do you turn text into a vector that captures its semantic meaning? Why does "king - man + woman = queen" work? Embeddings are the foundation of search, RAG, and recommendation systems — and they're simpler than you think.

← Previous

Ep 13: Batching & Concurrency

Next →

Ep 15: Embeddings

Next: Episode 15 — Embeddings

How does a computer know that 'king' is to 'queen' as 'man' is to 'woman'? The answer is embeddings.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.