Training vs Inference: How AI Learns and How It Responds

AI does ONE thing — predict the next word.

Your brain does the same thing. But better. Much better.

AI predicts the next word. One dimension. Just text.

Your brain predicts the next sound — a car is coming, you know before you see it. The next movement — a cricket ball is flying, your hand is already moving. The next emotion — your friend goes silent on a phone call for 2 seconds, and you know something is wrong. The next danger — you step back from the edge without thinking.

Both are prediction machines. AI does one type. You do all of them, simultaneously, on 20 watts.

But HOW does AI learn to predict? And how does it use that skill when you chat with it?

Two completely different operations

Training — how the model learns. Inference — how the model responds. Same model. Completely different processes, hardware, costs, and timescales.

Training: How the Model Learns

Training is where the magic happens — and where the billions of dollars go.

The Loop

The entire training process is one loop, repeated trillions of times:

Feed a sentence with the last word hidden: "The cat sat on the ___"

Model predicts using its current parameters: "banana" (wrong!)

Calculate the error — how far is "banana" from "mat"? This distance is called the loss function

Adjust the parameters — nudge all 5 trillion numbers slightly in the direction that would have made the answer closer to "mat." This is called gradient descent

Repeat with the next sentence

That's it. This loop runs trillions of times over months:

Data

Books, Wikipedia, GitHub, Reddit — trillions of words from the internet

Hardware

~50,000 GPUs running 24/7

Time

3–6 months

Cost

Estimated ~$1 billion for GPT-5

How Parameters Change

Remember from Episode 3 — parameters are just numbers. During training, they change with every step:

Each parameter gets a tiny nudge — gradient descent in action

Step 1: 0.4721 → Step 2: 0.4719

Step 1: -1.2830 → Step 2: -1.2850

Step 1: 0.0910 → Step 2: 0.0890

Each change is tiny — a fraction of a percent. But multiply that across 5 trillion parameters and trillions of training steps, and the model goes from predicting "banana" to predicting "mat."

Nobody programs intelligence

Nobody programs intelligence into the model. It emerges from this loop. Random noise in, useful predictions out. The model discovers grammar, facts, reasoning, and even humor entirely on its own — just by correcting billions of prediction errors.

Inference: How the Model Responds

When you open ChatGPT and type a message — that's inference.

Training is over. The parameters are frozen. Locked. Read-only. No learning is happening. The model is simply using what it already learned to predict the next token.

Token by Token

Inference works one token at a time:

Your prompt: "Write a professional email to reschedule a meeting"

The model predicts:

1. "Subject" → done, add it to the output
2. ":" → done, add it
3. "Request" → done
4. "to" → done
5. "Reschedule" → done
6. ...and so on, token by token

Why ChatGPT 'types'

This is why you see ChatGPT "typing" — it's not writing the whole answer and then showing you. It genuinely doesn't know word 2 until word 1 is done. Each word is a fresh prediction.

How Token Count Works

This is where it gets interesting. Every single token prediction requires the model to read all previous tokens.

Say your prompt is 50 tokens and the answer is 100 tokens:

Prediction Model reads Generates

The context grows with every word. That's why:

Longer conversations get slower — more to re-read each time

Long answers cost more — each token requires more compute

There's a context limit — 128K tokens max, then older messages get cut off

Input vs Output Tokens — Why They Cost Different

AI companies charge differently for input and output tokens:

Input Tokens Output Tokens

What Your prompt AI's answer

Output tokens cost 4x more

Output tokens cost 4x more because each one requires its own prediction pass AND re-reading all previous tokens. Input tokens are processed in a single parallel operation — much cheaper.

This is also why streaming exists. When you see ChatGPT showing you words one at a time, it's not a UI effect — the model is genuinely generating them one at a time. It can't show you the full answer instantly because it doesn't know the full answer until it's predicted every token.

The Key Difference: Parameters

This is the single most important distinction:

🔥 Training

CHANGE

Parameters adjust with every prediction step. That's how the model learns.

FROZEN 🔒

Parameters are read-only. The model uses what it learned but never updates.

Training Inference

Your conversations don't teach the model anything

When your chat ends, nothing was learned. The model is identical before and after your conversation. That's why ChatGPT doesn't "remember" you across sessions — inference is completely read-only.

The Full Picture — Interactive Diagram

The training loop (left) versus the inference pipeline (right) — side by side:

🔥 TRAINING — Learning Phase

📄 Training Data

Books, Wikipedia, GitHub, Internet

Trillions of words

📝 Feed a Sentence

"The cat sat on the ___"

❌ "banana"

→

✓ "mat"

📏 Calculate Error

How far is "banana" from "mat"? (Loss Function)

🔧 Adjust Parameters

Nudge 5 trillion numbers slightly

0.4721

→

0.4719

(Gradient Descent)

🔁 Repeat × TRILLIONS of times

Until "banana" becomes "mat"

REPEAT LOOP

Training Resources

50,000 GPUs · 3–6 months · ~$1 Billion

Parameters: ADJUSTING with every step

💬 Your Prompt

"Write a professional email to

reschedule a meeting"

🔒 Load Frozen Parameters

Same 5 trillion numbers from training

Read-only. Never change.

🔮 Predict Next Token

One token at a time:

"Subject" → ":" → "Request" → "to" → ...

next

token

📝 Output (token by token)

Subject: Request to Reschedule

Dear Mr. Sharma,

I would like to reschedule our

meeting to Friday...

Best regards, Abhishek Paul

No learning happened. No parameters changed.

Inference Resources

1–8 GPUs · Seconds · ~$0.002 per chat

Parameters: FROZEN 🔒 (read-only)

VS

Comparison

🔥 Training

What happens

Model LEARNS from data

Model RESPONDS to you

Parameters

Change every step

Frozen 🔒 (read-only)

GPUs

50,000

1–8

Time

3–6 months

Seconds

Cost

~$1 Billion

~$0.002

How often

Once

Millions / day

💡 Training = School (expensive, once)Inference = Exam (cheap, unlimited)

@making_in_time_ · Learn AI Infrastructure — Episode 4

The analogy that sticks

Training = School (expensive, done once, shapes who you are).
Inference = Taking an exam (cheap, unlimited, uses what you already learned).

The Brain Comparison

Your brain and AI are both prediction machines. But the comparison reveals how different they really are:

AI (GPT-5) Your Brain

AI has to re-read your entire conversation for every single word it generates. Your brain doesn't re-read your entire life every time you speak. That's a fundamental architectural difference — and it's why AI still has a long way to go.

The honest truth

AI training costs $1 billion and takes 6 months. Your brain was trained by dal chawal, mummy ki daant, and gully cricket. For free. On 20 watts. And it can feel, laugh, love, and know when to stay quiet. We're just getting started.

Practical Takeaways

Training and inference are completely different operations — different hardware, costs, and timescales

Parameters change during training, freeze during inference — that's the core difference

AI only predicts the next token — everything else (coding, reasoning, humor) emerges from this one skill

Output tokens cost more than input — because each requires a separate prediction pass

Your conversations don't teach the model anything — inference is read-only

Longer answers = more compute = more cost — every token re-reads the full context

The single most important thing

Training is where the model becomes intelligent. Inference is where it demonstrates that intelligence. One is expensive and happens once. The other is cheap and happens billions of times a day. Understanding this distinction unlocks everything else in AI engineering.

Next: Episode 5 — Context Window

What does "128K tokens" actually mean? Why does AI forget what you said 5 minutes ago? And why does your conversation suddenly get dumber halfway through? The answer is the context window — and it's the most practical concept in AI engineering.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.