Parameters: What "70 Billion" Actually Means

GPT-5 — estimated 2 to 5 trillion parameters. Claude Opus 4.6 — estimated 5 trillion parameters. Your brain — 100 trillion synapses.

GPT-5 is roughly 5% of your brain. And it passes the bar exam, writes production code, and speaks 100+ languages.

But what IS a parameter? Let's fix that.

The Simplest Explanation

A parameter is a single number that stores a tiny piece of learned knowledge.

That's it. One number. Like 0.0023 or -1.47 or 0.891.

When someone says "70 billion parameter model" — they mean 70 billion individual numbers, each one storing a small decision the model learned during training.

The Chai Analogy

Imagine you're building an AI to make the perfect cup of chai:

Version 1: 1 parameter

Just one number — "put 2 spoons of sugar." That's the entire model. One decision. It makes the same chai every time.

Version 2: 6 parameters

Sugar: 2.0 · Milk ratio: 0.3 · Tea leaves: 1.5 · Water: 200ml · Boil time: 4.0 · Temperature: 95°. A specific cup of chai, but still the same for everyone.

Version 3: 1,000 parameters

Adjusts based on inputs — who's drinking, time of day, weather, regional preference. Each parameter captures a small relationship: "when it's cold AND the person is from Kolkata, increase ginger by 0.3."

Version 4: 70 billion parameters

Understands language, culture, context, and nuance so deeply that you can describe what you want in natural language, and it figures out the perfect response.

The rule

More parameters = more nuance = more understanding. Each parameter captures a tiny learned relationship from training data.

What Parameters Actually Are: Weights

In technical terms, parameters are called weights. They live in the mathematical layers of the neural network.

From Episode 2, we learned that a transformer has multiple layers containing self-attention, feed-forward networks, and embedding layers. Every single number in every single matrix across all these layers — that's a parameter.

Where Do the 70 Billion Live?

Let's take Llama 3 70B as an example:

Component What It Does Parameters (approx.)

Each parameter is typically stored as a 16-bit floating point number (2 bytes). So the raw model file is:

70,000,000,000 × 2 bytes = 140 GB

That's why running a 70B model requires serious hardware.

You need enough memory to hold all 70 billion numbers at once.

How Parameters Get Their Values: Training

Here's the key insight: nobody writes these numbers by hand.

When a model is first created, all 70 billion parameters are set to random values. The model is essentially a newborn — it knows nothing.

Then comes training:

Show the model a sentence: "The cat sat on the ___"

The model, with its random parameters, guesses: "banana" (wrong)

Calculate the error (how far "banana" is from "mat")

Adjust the parameters slightly — nudge them toward the correct answer

Repeat with the next sentence

Repeat billions of times

Gradient descent — one tiny update at a time

Before update: parameter = 0.4721

After one step: parameter = 0.4719 ← tiny change: -0.0002

Multiply this across 70 billion parameters

× billions of training steps

= a model that can write code and explain quantum physics

Where does the training text come from?

Books (thousands of them)
Wikipedia (all languages)

Web pages (billions, filtered for quality)

Code repositories (GitHub, Stack Overflow)

Scientific papers, conversations, documentation

Llama 3 training data

Llama 3 was trained on approximately 15 trillion tokens of text — roughly 11 trillion words. More text than any human could read in thousands of lifetimes. Each parameter represents a pattern extracted from that ocean of text.

The Size-Intelligence Relationship

Does bigger always mean better? Mostly yes, but with diminishing returns.

Model Parameters What It Can Do

GPT-2 (2019) 1.5B Basic text completion, sometimes coherent

Try It Yourself — Interactive Demo

Drag the slider to grow the AI's brain. Watch what each model can and can't do:

The Scaling Laws

In 2020, OpenAI published scaling laws: predictable relationships between model size, training data, and performance.

Double the parameters → consistent improvement (but not double the performance)

Training data matters as much as size — a 7B model trained on 2T tokens can beat a 13B model trained on 500B tokens

Returns diminish — going from 7B to 70B is a huge jump; 70B to 700B is a smaller relative improvement

Smaller but smarter

Llama 3.1 8B today outperforms GPT-3 175B from 2020 — despite being 22× smaller — because it was trained on much more and better data. The industry has moved toward smaller, better-trained models.

Parameters and Memory: The VRAM Problem

If you want to run a model, you need to fit all its parameters in memory.

The Math

Memory required for a 70B model

FP32 (full precision):    4 bytes/param → 70B × 4   = 280 GB
FP16 (half precision):    2 bytes/param → 70B × 2   = 140 GB
INT8 (quantized):         1 byte/param  → 70B × 1   =  70 GB
INT4 (heavily quantized): 0.5 bytes     → 70B × 0.5 =  35 GB

+ KV cache (attention states): 5-20 GB

+ Runtime overhead: 2-4 GB

Real-World GPU Requirements

Model Size Quantization VRAM Needed GPU Options

Why quantization matters

Quantization is the art of making those 70 billion numbers smaller without losing too much intelligence. INT4 quantization shrinks a 70B model from 140 GB to ~35 GB — making it runnable on consumer hardware. We'll cover quantization in depth in a future episode.

Parameters vs. Hyperparameters

Don't confuse these — they're fundamentally different:

Parameters

70 billion of them. Learned during training. The model discovers them automatically. You never touch them. Examples: the 70B weight matrices in Llama 3.

Hyperparameters

A handful. Set by humans before training. Things you control: Learning rate · Number of layers · Hidden dimension · Attention heads

The inference knobs

When you adjust temperature or top-p in your API call, you're adjusting inference hyperparameters — knobs that control behavior. The 70 billion parameters stay completely fixed. They only change during fine-tuning (that's Module 7).

The Human Brain Comparison

People love comparing AI parameters to brain synapses. Let's be honest about it:

GPT-5's estimated 2-5 trillion parameters are roughly 2-5% of human brain synapses. And yet it passes bar exams, writes production code, and holds conversations in 100+ languages.

The misleading comparison

A synapse and a parameter work completely differently. A single biological synapse does more complex computation than a single neural network parameter. The architectures aren't comparable. The honest takeaway: We've achieved remarkable capability with a fraction of biology's complexity. GPT-5 and Claude Opus are doing things that would have seemed like science fiction 5 years ago — with just 5% of the connections in your head. And we're just getting started.

Practical Takeaways

"70B model" = 70 billion learned numbers — not magic, just math

Bigger models need more VRAM — calculate before you download

Size isn't everything — training data quality and quantity matter just as much

Quantization lets you shrink models — trading small accuracy for big memory savings

Parameters are fixed after training — when you use a model, those 70B numbers never change

The single most important thing

A parameter is just a number. 70 billion numbers, each one learned — not programmed. Nobody wrote them. The model figured them out by reading trillions of words. That's the entire magic.

Next: Episode 4 — Training vs Inference

How does a model learn (training) versus how does it respond (inference)? They're completely different processes with different hardware, costs, and bottlenecks. Understanding this distinction is crucial for anyone working with AI.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.