Test-Time Compute: Why 'Thinking' Models Are Different

Ask GPT-4 to solve a tricky math problem. It answers in 2 seconds. Wrong.

Ask o1 the same problem. It "thinks" for 30 seconds. Right.

What happened in those 30 seconds? The model didn't go look anything up. It didn't call an API. It literally thought harder — generating hundreds of internal reasoning tokens before giving you the final answer.

This is test-time compute — the idea that you can make a model smarter by letting it use more compute during inference. Not by training a bigger model. Not by giving it more data. Just by letting it think longer on your question.

And it's changing the economics of AI completely.

The Two Levers of AI Intelligence

Traditionally, making AI smarter meant one thing: train bigger models on more data. That's training-time compute — investing billions upfront to create a better model.

But there's a second lever: test-time compute — investing more compute when the model is actually answering your question.

Lever When Cost Example

The breakthrough insight: a smaller model with more test-time compute can beat a larger model with less. A 70B model that "thinks" for 60 seconds can outperform a 405B model that answers in 2 seconds — on reasoning tasks.

How Thinking Tokens Work

When o1 or DeepSeek-R1 "thinks," it generates a long chain of internal reasoning tokens before producing the final answer. These thinking tokens are:

Generated by the model — they're real tokens, not a special mechanism

Hidden from the user — you see "thinking..." but not the actual reasoning (usually)

Used as context — each thinking token informs the next, building up a solution step by step

What the Thinking Looks Like

User: "What is 37 × 48?"

Model's internal thinking tokens:

Let me solve 37 × 48 step by step.

I can break this down:

37 × 48 = 37 × 50 - 37 × 2

37 × 50 = 1850

37 × 2 = 74

1850 - 74 = 1776

Let me verify: 37 × 48

= (30 + 7) × 48

= 30 × 48 + 7 × 48

= 1440 + 336

= 1776 ✓

The answer is 1776.

Final answer to user: "37 × 48 = 1,776"

The user just sees "1,776" — but the model generated 100+ tokens of reasoning to get there. And those reasoning tokens cost compute and money.

🔧

Chain-of-Thought: The Foundation

Test-time compute builds on Chain-of-Thought (CoT) prompting, discovered in 2022. The observation: if you ask a model to "think step by step," it performs dramatically better on reasoning tasks.

Without CoT:

Q: "If a train travels 120 km in 2 hours, and then 180 km in 3 hours,

what's the average speed?"

A: "72 km/h" ❌ (just guessed)

With CoT:

Q: (same question, but add "think step by step")

A: "Total distance = 120 + 180 = 300 km

Total time = 2 + 3 = 5 hours
    Average speed = 300/5 = 60 km/h" ✅

CoT works because it forces the model to allocate compute to intermediate steps. Without CoT, the model must compute the answer in a single forward pass — one matrix multiply per token. With CoT, each reasoning step gets its own forward pass, dramatically increasing total compute.

Why More Tokens = More Compute

This is the key insight. Remember from Episode 4: each output token requires a full forward pass through the entire model. More tokens = more forward passes = more compute.

Standard answer: 5 tokens × 1 forward pass each = 5 forward passes

CoT answer: 50 tokens × 1 forward pass each = 50 forward passes

10x more compute → much better reasoning

The model is literally using its own output tokens as "scratch paper" for computation.

📊

o1: Scaling Test-Time Compute

OpenAI's o1 (September 2024) was the first model designed specifically to scale test-time compute. It was trained with reinforcement learning to generate effective reasoning chains.

How o1 Was Trained

Standard LLM training:

Input → Model → Output → Compare with ground truth → Update weights

o1's training (simplified):

Input → Model → Generate long reasoning chain → Get final answer

→ Was the answer correct? → Reward signal → Update via RL

→ Also: was the reasoning efficient? → Process reward → Update via RL

The key difference: o1 wasn't just trained to get the right answer. It was trained to think productively — to generate reasoning steps that actually help reach the correct conclusion. Bad reasoning gets penalized. Circular reasoning gets penalized. Productive reasoning gets rewarded.

Scaling Behavior

The remarkable finding: o1's performance scales predictably with thinking time:

Thinking Time AIME Math Score Improvement

More thinking = better answers, with a smooth, predictable curve. This is a scaling law for inference — analogous to the training scaling laws we've known about for years.

DeepSeek-R1: Open-Source Reasoning

DeepSeek-R1 (January 2025) showed that test-time compute models could be built without OpenAI's proprietary techniques. And they open-sourced everything.

The Remarkable Discovery

DeepSeek found that if you train a model with pure reinforcement learning on reasoning tasks (no supervised CoT data), the model spontaneously develops reasoning behaviors:

Training signal: Only "was the final answer correct?"

No examples of reasoning chains provided.

Result: The model learns to:

• Break problems into steps
• Try multiple approaches
• Check its own work
• Backtrack when stuck
• Verify answers before committing

The model invented chain-of-thought reasoning on its own, just from the reward signal of correct answers. It even developed "aha moments" — moments where it realizes its approach is wrong and pivots:

Let me try approach A...

Hmm, that gives me 847, which seems too large.

Wait, I think I made an error in step 3.

Let me reconsider...

Actually, I should use approach B instead.

...

Yes, approach B gives 127, which makes much more sense.

R1's Architecture

Base model: DeepSeek-V3 (671B MoE, 37B active)

Training: Multi-stage

Stage 1: Cold start with a few thousand CoT examples

Stage 2: Large-scale RL with GRPO (Group Relative Policy Optimization)

Stage 3: Rejection sampling + supervised fine-tuning

Stage 4: Final RL for helpfulness + safety

Distillation: DeepSeek also showed you can distill R1's reasoning into smaller models. R1-distilled versions at 7B, 14B, and 32B parameters still show strong reasoning — they learned the pattern of productive thinking from the larger model.

The Economics of Thinking

Test-time compute fundamentally changes AI pricing:

Token Costs

Standard query (GPT-4):

Input: 500 tokens × $10/M = $0.005

Output: 200 tokens × $30/M = $0.006

Total: $0.011

Thinking query (o1):

Input: 500 tokens × $15/M = $0.0075

Thinking: 5,000 tokens × $60/M = $0.30 ← THIS IS THE EXPENSIVE PART

Output: 200 tokens × $60/M = $0.012

Total: $0.32

A thinking query costs 30x more than a standard query. But for hard problems (math, code, analysis), it's worth it because the answer is actually correct.

When to Use Thinking Models

The rule: Use thinking models for tasks with verifiable correct answers. Math, logic, code, analysis. Don't use them for creative writing or simple lookups.

🔬

Process Reward Models (PRMs)

A key enabler of test-time compute is the Process Reward Model — a separate model that evaluates each step of reasoning, not just the final answer.

Outcome vs Process Reward

Outcome Reward Model (ORM):

"Your final answer is correct" → reward = 1.0
"Your final answer is wrong" → reward = 0.0

No feedback on intermediate steps.

Process Reward Model (PRM):

Step 1: "120 + 180 = 300" → reward = 1.0 (correct step)
Step 2: "300 / 2 = 150" → reward = 0.0 (wrong step! should be /5)
Step 3: "Speed = 150 km/h" → reward = 0.0 (wrong, follows from bad step)

PRMs enable:

Better RL training — reward good reasoning, not just lucky answers

Search during inference — generate multiple reasoning paths, use PRM to pick the best one

Early termination — stop a bad reasoning chain before wasting more tokens

🛡️

Search Strategies at Inference

With enough test-time compute, you can search for the best answer:

Best-of-N Sampling

Generate N complete solutions, pick the one the PRM scores highest:

Query: "Solve this optimization problem"

Solution 1: [reasoning...] → Answer: 42 → PRM score: 0.85

Solution 2: [reasoning...] → Answer: 37 → PRM score: 0.45

Solution 3: [reasoning...] → Answer: 42 → PRM score: 0.92 ✓ ← Pick this one

Solution 4: [reasoning...] → Answer: 45 → PRM score: 0.31

Tree Search

Instead of generating complete solutions, explore a tree of reasoning steps:

Start → Step A1 (PRM: 0.9) → Step A2 (PRM: 0.85) → Answer (correct ✓)

→ Step B1 (PRM: 0.3) → pruned (bad start)
     → Step C1 (PRM: 0.7) → Step C2 (PRM: 0.4) → pruned (went wrong)

This is more compute-efficient than best-of-N because bad paths get pruned early.

📦

The Scaling Laws of Inference

The most important finding from test-time compute research:

Training compute and inference compute are partially substitutable.

A smaller model with more inference compute can match a larger model with less:

Equivalent performance:

405B model × 1 second thinking ≈ 70B model × 60 seconds thinking

The smaller model "thinks harder" to compensate for fewer parameters.

This has profound implications:

Cheaper deployment — serve a 70B model instead of 405B

Pay per difficulty — easy questions get fast answers, hard questions get more thinking

Adaptive compute — the model itself can decide how hard to think

🚀

Practical Takeaways

Test-time compute = spending more inference compute on hard problems — thinking tokens are real tokens that cost money

Chain-of-thought is the mechanism — each reasoning token is a forward pass, giving the model more compute

Performance scales with thinking time — more thinking = better answers, predictably

Use thinking models for verifiable tasks — math, code, logic — not creative writing or simple lookups

Thinking queries cost 10-30x more — but correctness on hard problems makes it worth it

Training and inference compute are substitutable — smaller models can compensate with more thinking

🔄

What's Next?

Episode 63: What's Next in AI — Agents, world models, embodied AI, the reasoning frontier. Where is this all heading? Are we hitting scaling law plateaus? We'll look at the cutting edge and the next 3-5 years of AI.

← Previous

Ep 61: State Space Models

Next →

Ep 63: What's Next in AI

Next: Episode 63 — What's Next in AI

You now understand how modern AI actually works — not as a user, but as an engineer. So where is all of this heading?

This is part of a 98-episode series covering AI engineering from tokens to production deployment.