Scaling Laws: Why Bigger Isn't Always Better

OpenAI spent an estimated $1 billion training GPT-5. Google poured similar resources into Gemini Ultra. Anthropic raised billions to train Claude.

The question nobody was asking until 2022: were they spending that money optimally?

The answer, it turns out, was no. Not even close.

Scaling laws are mathematical relationships that predict how model performance improves as you increase three things: parameters (model size), data (training tokens), and compute (GPU hours). They're the physics of AI training — and they revealed that the entire industry was building models wrong.

The Original Scaling Laws (Kaplan et al., 2020)

OpenAI published the first major scaling laws paper. Their finding was elegant:

Model performance improves as a power law with each of three factors:

Loss ∝ N^(-0.076)    # N = number of parameters
Loss ∝ D^(-0.095)    # D = number of training tokens
Loss ∝ C^(-0.050)    # C = compute budget (FLOPs)

In plain English:

10x more parameters → loss decreases by ~40%

10x more data → loss decreases by ~45%

10x more compute → loss decreases by ~30%

The seductive conclusion: Just make models bigger! Loss keeps going down! This launched the "bigger is better" era. GPT-2 had 1.5B parameters. GPT-3 had 175B. People were planning trillion-parameter models.

But there was a problem hiding in the math.

Chinchilla: The Paper That Changed Everything

In March 2022, DeepMind published what might be the most important AI paper of the decade: "Training Compute-Optimal Large Language Models" — known as the Chinchilla paper.

Their key insight: OpenAI's scaling laws had a blind spot. They answered "how does performance scale with size?" but not "given a fixed compute budget, what's the OPTIMAL split between model size and training data?"

DeepMind trained over 400 models of different sizes on different amounts of data. The answer was shocking:

For compute-optimal training, parameters and training tokens should scale equally.

The formula:

Optimal tokens ≈ 20 × number of parameters

Model Parameters Optimal Training Tokens Optimal Training Data

The bombshell: GPT-3 (175B parameters) was trained on only 300B tokens. According to Chinchilla, it should have been trained on 3.5 TRILLION tokens. GPT-3 was massively undertrained.

DeepMind proved this by training a 70B parameter model (Chinchilla) on 1.4T tokens. Despite being 2.5x smaller than GPT-3, it outperformed GPT-3 on virtually every benchmark.

Chinchilla (70B, 1.4T tokens) > GPT-3 (175B, 300B tokens)

Smaller model + more data = better performance + cheaper inference

This was a paradigm shift. The industry had been wasting billions of dollars making models too large and training them on too little data.

🔧

Why This Matters Economically

The Chinchilla insight has massive financial implications:

Training cost stays the same...

A 70B model trained on 1.4T tokens uses roughly the same compute as a 175B model trained on 300B tokens. Same GPU hours, same electricity bill.

...but inference gets WAY cheaper

GPT-3 Style Chinchilla Style

You get a better model that costs less to serve. That's not a tradeoff — that's a free lunch.

This is why Llama 2 (70B) was trained on 2T tokens. Why Llama 3 (8B) was trained on 15T tokens — nearly 2000× the Chinchilla-optimal ratio! The industry went from "undertrained" to "massively overtrained" because inference cost matters more than training cost at scale.

📊

Beyond Chinchilla: The Inference-Optimal Frontier

The Chinchilla ratio (20:1 tokens-to-parameters) optimizes for training compute. But in the real world, you train once and serve millions of requests. So the real optimization is:

Total cost = training cost + (inference cost × number of requests)

If your model serves billions of requests, it's worth spending MORE on training (using more data) to get a SMALLER model that's cheaper to serve:

Scenario A: 70B model, Chinchilla-optimal

Training: $10M

Inference: $0.01/request × 1B requests = $10M

Total: $20M

Scenario B: 8B model, overtrained (15T tokens)

Training: $15M (more data = more compute)

Inference: $0.001/request × 1B requests = $1M

Total: $16M ← CHEAPER overall

This is why Llama 3 8B was trained on 15T tokens. Meta knew it would be deployed everywhere — phones, laptops, edge devices. They overtrained the small model deliberately to make inference cheap.

The Diminishing Returns Curve

Scaling laws follow a power law, which means diminishing returns are baked in:

Going from 1B → 10B parameters: ~40% loss reduction

Going from 10B → 100B parameters: ~40% loss reduction

Going from 100B → 1T parameters: ~40% loss reduction

Each 10× increase gives the same percentage improvement. But the COST scales linearly (or worse):

Training 1B model: $10K

Training 10B model: $100K (10× cost for 40% improvement)

Training 100B model: $1M (100× cost for another 40%)

Training 1T model: $100M (10000× cost for another 40%)

At some point, the cost of the next increment of quality exceeds its value. We're arguably there already for many applications.

The practical implication: For most real-world use cases, a well-trained 8B model with good fine-tuning beats a 70B model out of the box. Scaling has limits.

Emergent Abilities: The Counterargument

There's a fascinating complication. Some capabilities don't improve gradually — they appear suddenly at certain scales:

Chain-of-thought reasoning: Barely works at 10B, works well at 60B+

Few-shot learning: Random at 1B, useful at 10B, impressive at 100B+

Code generation: Poor at 7B, competent at 34B, excellent at 70B+

Multi-step math: Basically zero at small scales, then jumps at ~50B+

These emergent abilities are the argument for still building large models. You can't get GPT-4-level reasoning from a 7B model no matter how much data you throw at it. Some capabilities require a minimum model size to emerge.

But recent research suggests many "emergent abilities" might be artifacts of how we measure them (using metrics with sharp thresholds rather than gradual ones). The debate continues.

🔬

Scaling Laws for Your Projects

You probably aren't training a foundation model. But scaling laws still matter for you:

For fine-tuning:

More high-quality fine-tuning data > more training epochs on small data

If your fine-tune is overfitting, don't increase model size — increase data quality and quantity.

For RAG:

Better retrieval on existing data > ingesting 10× more data

Scaling your vector store from 1M to 10M chunks won't help if your retrieval quality is bad.

For inference:

Smaller model + better prompting > bigger model + basic prompting

Before upgrading from 8B to 70B, try: better system prompts, few-shot examples, chain-of-thought. You'd be surprised.

For choosing a model:

Your Use Case Recommended Size Why

🛡️

The Future of Scaling

Three trends are reshaping scaling laws:

1. Data walls. We're running out of high-quality internet text. The Chinchilla recipe says a 1T parameter model needs 20T tokens. But high-quality English text on the internet might only be 5-10T tokens. This is driving the synthetic data revolution (next episode!).

1. Mixture of Experts (MoE). Instead of one massive model, use many smaller "expert" models and route each request to the relevant ones. Mixtral (8×7B) has 47B total parameters but only activates 13B per token. Better scaling at the same inference cost.

1. Test-time compute. Instead of scaling the model (training compute), scale the thinking (inference compute). This is what OpenAI's o1 does — it "thinks longer" on hard problems. A smaller model that reasons for 30 seconds can beat a larger model that answers instantly.

The scaling game is shifting from "bigger models" to "smarter architectures" and "smarter inference."

📦

Practical Takeaways

Chinchilla's rule: tokens ≈ 20× parameters — for compute-optimal training, don't make models bigger without proportionally more data

Inference cost > training cost — in production, you'll spend more serving the model than training it

Overtrain small models for deployment — Meta trained Llama 3 8B on 15T tokens for a reason

Diminishing returns are real — each 10× in compute gives the same percentage improvement, not the same absolute improvement

Start small, scale only if needed — a fine-tuned 8B model beats a generic 70B model for most tasks

The future is smarter, not bigger — MoE, test-time compute, and better data matter more than raw scale

🚀

What's Next?

Episode 90: Synthetic Data — We're running out of internet text. The solution? Have AI generate its own training data. But there's a dark side — model collapse, echo chambers, and the risk of AI eating its own tail. We'll explore self-instruct, Alpaca, and why synthetic data is both the future and a ticking time bomb.

← Previous

Ep 88: Data Pipelines for AI

Next →

Ep 90: Synthetic Data

Next: Episode 90 — Synthetic Data

The internet has 5-10 trillion tokens of quality text. We're running out of human-written data. Synthetic data is the answer — and the risk.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.