Temperature, Top-p, Top-k: The Knobs That Control Creativity

Ask ChatGPT "Write me a poem about rain" ten times. You'll get ten different poems.

Ask it "What is 2+2?" ten times. You'll get "4" every single time.

Same model. Same parameters. Same everything. So why do answers sometimes vary and sometimes don't?

Three settings control this: Temperature, Top-p, and Top-k. They're in every API call, most people ignore them, and they dramatically change how the AI behaves.

How the Model Actually Picks a Word

Before we touch the knobs, you need to understand what's happening under the hood.

From Episode 4, you know the model predicts one token at a time. But it doesn't predict a single token — it predicts a probability distribution over ALL possible tokens.

For the prompt "The capital of France is ___":

Paris: 92.4%

Lyon: 1.8%

the: 1.2%

Berlin: 0.9%

a: 0.8%

France: 0.5%

...

(50,000+ tokens with tiny probabilities)

The model is 92.4% confident the next word is "Paris." Now it has to pick one. How it picks is where our three knobs come in.

Temperature: The Chaos Dial

Temperature controls how "sharp" or "flat" the probability distribution is.

Low temperature (0.0 - 0.3): Makes high-probability tokens MORE dominant. The model becomes predictable, deterministic, boring (in a good way for factual tasks).

High temperature (0.7 - 1.5): Flattens the distribution. Lower-probability tokens get a real chance. The model becomes creative, unpredictable, sometimes unhinged.

The Math (Simple Version)

adjusted_probability = original_probability^(1/temperature)

Let's see what happens to our "capital of France" example:

Token Original Temp=0.1 Temp=0.5 Temp=1.0 Temp=2.0

At temperature 0.1, "Paris" gets 99.99% — essentially guaranteed. At temperature 2.0, "Paris" drops to 72% and random tokens get a real shot. That's when you get weird, creative (or nonsensical) outputs.

Temperature = 0

Special case. Temperature 0 means greedy decoding — always pick the highest probability token. No randomness at all. Every run gives the identical output.

python

OpenAI API

response = client.chat.completions.create(
model="gpt-4",
    messages=[{"role": "user", "content": "What is 2+2?"}],
    temperature=0  # Always picks highest probability token

)

🔧

Top-k: Only Consider the Top K Options

Top-k is simpler. Before sampling, throw away everything except the top K tokens. Then sample from those.

Top-k = 3 for "capital of France":

Before: Paris(92.4%), Lyon(1.8%), the(1.2%), Berlin(0.9%), ...50,000 more

After: Paris(95.3%), Lyon(1.9%), the(1.2%) ← renormalized to 100%

Everything else = 0% chance

Top-k Effect Use Case

The problem with top-k: it's a fixed number regardless of context. Sometimes the model is 99% sure — top-k=50 forces it to consider 49 garbage tokens. Sometimes it's genuinely torn between 200 options — top-k=50 cuts off valid choices.

That's where top-p comes in.

📊

Top-p (Nucleus Sampling): The Smart Cutoff

Top-p dynamically adjusts how many tokens to consider based on cumulative probability.

Top-p = 0.9 means: Keep adding tokens (from highest to lowest probability) until their combined probability reaches 90%. Everything else gets cut.

Example 1: Model is confident

"Capital of France is ___"

Paris: 92.4% → cumulative: 92.4% ← Already past 0.9!

Result: Only "Paris" is considered. top-p=0.9 → 1 token.

Example 2: Model is uncertain

"A good programming language is ___"

Python: 18% → cumulative: 18%

JavaScript: 15% → cumulative: 33%

Rust: 12% → cumulative: 45%

Go: 10% → cumulative: 55%

TypeScript: 9% → cumulative: 64%

Java: 8% → cumulative: 72%

C++: 7% → cumulative: 79%

C#:          6%  → cumulative: 85%

Kotlin: 5% → cumulative: 90% ← Hit threshold!

Result: 9 tokens considered. top-p=0.9 → 9 tokens.

Top-p is adaptive. When the model is sure, it picks from fewer options. When it's uncertain, it considers more. This is why top-p is generally preferred over top-k.

How They Work Together

These aren't mutually exclusive. You can combine them, and they're applied in order:

Step 1: Model produces raw probabilities for all ~50,000 tokens

Step 2: Apply temperature → sharpen or flatten the distribution

Step 3: Apply top-k → keep only top K tokens

Step 4: Apply top-p → keep tokens until cumulative probability = p

Step 5: Sample one token from the remaining candidates

Common Combinations

Use Case Temperature Top-p Top-k Why

The OpenAI Default

OpenAI's default is temperature=1.0, top_p=1.0 — both neutral. This means the raw probability distribution is used as-is. For most applications, you want to change at least one.

Pro tip: OpenAI recommends changing temperature OR top_p, not both. Adjusting both can have unpredictable interactions.

Practical Settings by Task

Here's what I actually use in production:

python

Code generation — I want correctness

response = client.chat.completions.create(
model="gpt-4",
    messages=messages,
    temperature=0,        # Deterministic
    # top_p default 1.0

)

Customer support chatbot — natural but not wild

response = client.chat.completions.create(
model="gpt-4",
    messages=messages,
    temperature=0.3,      # Mostly consistent
    top_p=0.9,            # Cut long-tail nonsense

)

Marketing copy — creative but coherent

response = client.chat.completions.create(
model="gpt-4",
    messages=messages,
    temperature=0.8,      # Creative
    top_p=0.95,           # Still somewhat focused

)

Ollama equivalent

ollama run llama3.1 --temperature 0.3 --top-p 0.9 --top-k 40

🔬

The "Temperature 0 Isn't Actually Deterministic" Gotcha

Here's something that trips up engineers: temperature=0 with GPT-4 sometimes gives different outputs across calls.

Why? GPU floating-point operations aren't perfectly deterministic across different machines. Your request might hit different servers. The math rounds slightly differently, and occasionally a different token wins.

OpenAI added seed parameter to address this:

python

response = client.chat.completions.create(
model="gpt-4",
    messages=messages,
    temperature=0,
    seed=42  # Now deterministic (mostly)

)

Even with seed, they only guarantee "best effort" determinism. For truly reproducible outputs, you need to run your own model locally.

🛡️

Try It Yourself — Temperature Explorer

python

from openai import OpenAI

client = OpenAI()
prompt = "Write a one-sentence story about a robot."

for temp in [0, 0.3, 0.7, 1.0, 1.5]:

responses = []
    for _ in range(3):
        r = client.chat.completions.create(
            model="gpt-4",
            messages=[{"role": "user", "content": prompt}],
            temperature=temp,
            max_tokens=50
        )
        responses.append(r.choices[0].message.content)

print(f"\n--- Temperature {temp} ---")
    for i, resp in enumerate(responses):
        print(f"  Run {i+1}: {resp}")

Run this and watch: at temp=0, all three responses are identical. At temp=1.5, they're wildly different. That's the entire concept in action.

📦

Practical Takeaways

Temperature controls randomness — 0 = deterministic, 1+ = creative, 1.5+ = chaotic

Top-k fixes the number of candidates — simple but inflexible

Top-p adapts to the model's confidence — preferred over top-k in most cases

Use temperature=0 for factual/code tasks — correctness over creativity

Use temperature=0.7-0.9 for creative tasks — variety without nonsense

Don't change temperature AND top_p simultaneously unless you know what you're doing

🚀

What's Next?

Episode 7: Why LLMs Hallucinate — The model just confidently told you a fake Supreme Court case citation. It wasn't lying — it was doing exactly what it's designed to do. Why probability-based generation makes hallucination inevitable, and what you can do about it.

← Previous

Ep 5: Context Window

Next →

Ep 7: Why LLMs Hallucinate

Next: Episode 7 — Why LLMs Hallucinate

A lawyer used ChatGPT to write a legal brief. It cited six court cases. None of them existed. Here's why LLMs make things up.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.