Why LLMs Hallucinate: Confidence Without Correctness

A lawyer used ChatGPT to write a legal brief. It cited six court cases. The judge checked. None of them existed. Not one. Made up case names, made up citations, made up rulings. The lawyer got sanctioned.

ChatGPT didn't "lie." It didn't "make a mistake." It did exactly what it's designed to do — predict the next most likely token. And the most likely tokens after "In the landmark case of" are tokens that look like a real case citation.

This is hallucination. And it's not a bug. It's a fundamental feature of how these models work.

What Is Hallucination?

Hallucination is when an LLM generates content that is fluent, confident, and wrong. Not typos. Not misunderstandings. Completely fabricated information delivered with the same tone as verified facts.

Examples:

What you asked What the model said Reality

The dangerous part isn't that it's wrong. It's that it sounds exactly like when it's right. There's no hesitation, no "I'm not sure," no change in confidence. The model doesn't know it's hallucinating.

Why It Happens: The Probability Problem

Go back to Episode 6. The model generates one token at a time based on probability. It doesn't have a fact database. It doesn't "look up" answers. It asks: "Given all the text I've seen during training, what token is most likely to come next?"

That's it. That's the entire mechanism.

The Fundamental Mismatch

What you think the model does:

Question → Look up answer → Respond

What the model actually does:

Question → What tokens typically follow this pattern? → Respond

When you ask "Who won the 1983 Cricket World Cup?", the model doesn't check a database. It recognizes the pattern — text about the 1983 World Cup in its training data overwhelmingly mentions "India" and "Kapil Dev." So those tokens have high probability. Correct answer, but for the wrong reason.

When you ask "Who won the 1987 Ranji Trophy semifinals?", the model has seen much less text about this. It still needs to produce confident, fluent text — that's what it was trained to do. So it assembles plausible-sounding tokens. A real team name. A real player name. A plausible score. All wrong.

It's Not Retrieving — It's Generating

Imagine someone who has read every book in a library but can't go back and check them. They can only recall patterns — what kind of answer typically follows this kind of question. Most of the time, the pattern matches reality. Sometimes it generates a plausible answer from a pattern that doesn't correspond to any real fact.

That's hallucination.

🔧

The Five Types of Hallucination

1. Fabricated Facts

The model states something that sounds specific and real but is entirely invented.

User: "What year was the Mumbai-Pune Expressway completed?"

Model: "The Mumbai-Pune Expressway was completed in 2002."

Reality: It was opened in 2002, but construction continued until 2003.

Close enough? Or a subtle hallucination?

Factual hallucinations exist on a spectrum. Sometimes it's close but slightly off. Sometimes it's completely fabricated. The model treats both with equal confidence.

1. Fabricated Citations

This is the lawyer problem. The model generates academic-looking references:

"According to Zhang et al. (2023), 'Efficient KV Cache Compression

for Long-Context Inference,' published in NeurIPS..."

This might be a real paper. Or the model might have assembled a plausible author name + plausible title + plausible venue. You can't tell without checking.

1. False Confidence on Ambiguous Questions

User: "Is it better to use Rust or Go for microservices?"

Model: "Rust is definitively better for microservices because..."

There is no definitive answer. But the model generates one because it was trained on text where people state opinions confidently.

1. Conflation

The model merges information from different sources:

User: "Tell me about the Salar Jung Museum in Hyderabad"

Model: "The Salar Jung Museum, founded in 1951, houses the Koh-i-Noor

diamond among its collections..."

The museum is real (founded 1951). The Koh-i-Noor is real. But it's in the Tower of London, not the Salar Jung Museum. The model conflated two facts about Indian heritage.

1. Instruction Hallucination

The model adds content you didn't ask for — or ignores constraints:

User: "Translate this to French. Don't add any commentary."

Model: "Here's the translation: [French text]

Note: I noticed a grammatical error in your original..."

It couldn't help itself. The pattern of being "helpful" overrode the explicit instruction.

📊

When Hallucination Gets Worse

Certain conditions make hallucination more likely:

Condition Why It Increases Hallucination

The pattern is clear: the less training data exists for a topic, and the more specific you ask the model to be, the more it hallucinates.

Can the Model "Know" It's Wrong?

No. And here's why.

The model has no internal fact-checking mechanism. It has no way to distinguish between:

A token it generated because it memorized a fact from training data

A token it generated because it matches a plausible pattern

Both feel the same to the model. Both come from the same process — the softmax probability over the vocabulary. There's no "confidence" signal separate from the token probability.

Some research suggests that models DO have internal representations that correlate with factual accuracy — but they don't use these during generation. It's like knowing the stove is hot but touching it anyway because the next action in the sequence is "touch stove."

Mitigation Strategies That Actually Work

1. Retrieval-Augmented Generation (RAG)

Instead of relying on the model's memory, give it the source material:

python

Bad: "What's our refund policy?"

Model: generates plausible-sounding policy from training data

Good: Retrieve actual policy, put it in context

context = vector_db.search("refund policy")
prompt = f"""Based ONLY on this document, answer the question.

Document: {context}

Question: What's our refund policy?

If the answer isn't in the document, say "I don't know."

"""

RAG doesn't eliminate hallucination, but it anchors the model to real source material. We'll cover RAG in depth in Module 4.

1. Temperature = 0 for Factual Tasks

From Episode 6 — low temperature means the model picks the highest-probability token every time. Less creative, but less likely to hallucinate.

python

For factual questions

response = client.chat.completions.create(
model="gpt-4",
    messages=messages,
    temperature=0  # Most probable tokens only

)

1. Explicit "I Don't Know" Permission

Models hallucinate partly because they're trained to be helpful. Give them permission to not answer:

System prompt: "If you're not confident in an answer, say

'I'm not sure about this — please verify.' Never fabricate

citations, statistics, or specific dates."

This doesn't guarantee compliance, but it significantly reduces confident fabrication.

1. Chain-of-Thought for Reasoning

For multi-step problems, force the model to show its work:

Instead of: "What's the answer?"

Ask: "Think step by step. Show your reasoning.

Then give the final answer."

When the model must show intermediate steps, errors become visible instead of hidden inside a confident final answer.

1. Self-Verification

Ask the model to check its own work:

Step 1: Generate answer

Step 2: "Review your answer. Are there any claims that might

be incorrect? List any statements you're uncertain about."

This catches some hallucinations — the model will sometimes flag its own fabrications when explicitly asked to review. Not foolproof, but adds a layer.

🔬

The Hard Truth

Hallucination cannot be completely eliminated in autoregressive language models. It's not a bug to fix — it's a mathematical consequence of the architecture.

The model predicts the next token based on probability. "Correctness" isn't a variable in that equation. "Most likely to follow this pattern" is. When those two align, you get brilliant answers. When they don't, you get confident nonsense.

Every mitigation strategy is a workaround. RAG works because you shift from "remember" to "read." Verification works because you add a checking step. But the underlying generation mechanism never "knows" if it's right.

The safest approach: Treat LLM output like a smart intern's first draft. Probably good, might have errors, definitely needs review for anything that matters.

🛡️

Practical Takeaways

Hallucination is inherent to how LLMs work — they predict tokens, not facts

The model doesn't know it's wrong — there's no internal fact-checking

Obscure topics + specific details = maximum hallucination risk — be cautious with niche questions

Use RAG for factual grounding — give the model source material instead of relying on memory

Temperature=0 reduces but doesn't eliminate hallucination — still pattern-matching, just less randomly

Always verify critical information — citations, numbers, medical/legal advice, anything with consequences

📦

What's Next?

Episode 8: CPU vs GPU Inference — Your laptop has 8 CPU cores and 16GB RAM. Your phone has 6 cores and 8GB RAM. Can they run AI models? When do you actually need a $40,000 GPU? The answer isn't what most people think.

← Previous

Ep 6: Temperature, Top-p, Top-k

Next →

Ep 8: CPU vs GPU Inference

Next: Episode 8 — CPU vs GPU Inference

A friend bought an RTX 4090 for ₹1,60,000. His wife thought it was for gaming. It wasn't. He runs Llama 3.1 70B on it.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.