Evaluation Frameworks: How Do You Know If Your Model Is Good?

"Our model achieves 89.3% on MMLU."

Sounds impressive, right? But what does it actually mean? Does it mean the model is smart? Useful? Safe? Ready for production?

Nope. It means the model can answer 89.3% of multiple-choice trivia questions correctly. That's it. It tells you almost nothing about whether your users will like it, whether it'll hallucinate in production, or whether it can actually do the thing you need it to do.

AI evaluation is broken. Benchmarks are gamed, leaderboards are manipulated, and the metrics that matter most — "does it actually work for my use case?" — are the hardest to measure.

Let's talk about what exists, what's useful, and what's absolute nonsense.

The Big Benchmarks

MMLU (Massive Multitask Language Understanding)

What it is: 15,908 multiple-choice questions across 57 subjects — from astronomy to law to computer science.

Example question:

Subject: Abstract Algebra

Which of the following is a true statement about the group Z_4

under addition modulo 4?

A) It is not cyclic

B) It has exactly two generators

C) Every element has order 4

D) It is isomorphic to the Klein four-group

Answer: B

Why it's popular: It's comprehensive, standardized, and easy to run. Every model reports MMLU.

Why it's problematic:

Multiple-choice ≠ real-world usage (nobody gives their chatbot A/B/C/D options)

Questions are from academic exams — heavily biased toward Western, English-language education

Models can score high by learning test-taking patterns, not by understanding

Some questions are ambiguous or have incorrect "correct" answers

It's been contaminated. Many models have been trained on MMLU questions or similar ones

MMLU score: 89.3% → "The model knows things"

Real question: "What should I do about this error in my React code?"

Model: hallucinates a nonexistent function

These metrics don't correlate as well as you'd think.

HumanEval (Code Generation)

What it is: 164 Python programming problems. The model writes code, and it's tested against unit tests.

python

Example HumanEval problem

def has_close_elements(numbers: List[float], threshold: float) -> bool:
"""Check if in given list of numbers, are any two numbers
    closer to each other than given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0], 0.3)
    True
    """
    # Model generates the implementation
    # Unit tests verify correctness

Why it's useful: Code is verifiable. It either passes the tests or it doesn't. No subjective judgment.

Why it's limited:

Only 164 problems — tiny sample size

All Python, all LeetCode-style

Doesn't test real-world coding (debugging, architecture, reading existing code)

pass@1 (get it right first try) vs pass@k (get it right in k attempts) — huge difference

Models are now specifically trained to ace HumanEval

LMSYS Chatbot Arena (ELO Rankings)

What it is: The most trusted benchmark in AI right now. Real users submit prompts. Two anonymous models respond. The user picks the better response. Over millions of comparisons, models get ELO ratings — the same system used in chess.

User sees:

┌─────────────────┬─────────────────┐

│ Model A (hidden) │ Model B (hidden) │

│ [response] │ [response] │

│ │ │

│ 👆 or 👆 │ │

│ "Which is better?" │

└─────────────────┴─────────────────┘

After choosing → Model identities revealed

Why it's the gold standard:

Real users, real prompts, real preferences

Blind comparison eliminates brand bias

ELO system is battle-tested (works for chess, works for AI)

Harder to game than multiple-choice benchmarks

Huge sample size (millions of votes)

Current rankings (as of writing):

Rank Model ELO

Limitations:

Users prefer verbose, confident responses (even if wrong)

Creative/writing tasks dominate — underrepresents code, math, reasoning

Self-selection bias — LMSYS users aren't representative of all users

Models can be optimized for "Arena vibes" (verbose, friendly, detailed)

The Problem with Benchmarks

All benchmarks share fundamental flaws:

1. Goodhart's Law

"When a measure becomes a target, it ceases to be a good measure."

The moment companies optimize for a benchmark, it stops measuring what it was designed to measure. Models are now specifically trained on MMLU-style questions. HumanEval has been memorized. Even Arena can be gamed with verbose, people-pleasing responses.

1. Benchmark Contamination

Training data for large models includes billions of web pages. Many benchmark questions are ON those web pages. The model might be "recalling" answers, not "reasoning" about them.

Is the model solving the math problem?

Or did it memorize the solution from its training data?

If MMLU questions appear in Common Crawl (they do),

every model trained on web data has potentially seen them.

1. The Gap Between Benchmarks and Real Use

Benchmark says Reality

🔧

Vibe Eval: The Human Judgment Layer

Increasingly, AI labs are admitting that numbers alone don't capture model quality. Enter vibe eval — structured human evaluation of model "vibes."

This isn't as unscientific as it sounds. It's actually how most production teams evaluate models:

python

Structured vibe eval rubric

eval_dimensions = {
"helpfulness": "Does it actually answer the question?",
    "accuracy": "Are the facts correct?",
    "tone": "Is it natural, not robotic?",
    "conciseness": "Does it get to the point?",
    "safety": "Does it refuse harmful requests appropriately?",
    "instruction_following": "Does it do what was asked?",
    "creativity": "For creative tasks — is it interesting?",

}

Rate each dimension 1-5 for a sample of 100 prompts

Compare candidate model vs production model

Statistical significance test on the ratings

How to do vibe eval well:

Use blind comparison — evaluators don't know which model is which

Use your actual users' prompts — not synthetic test cases

Rate multiple dimensions — not just "which is better overall?"

Use enough samples — 100+ comparisons minimum for statistical power

Include edge cases — the prompts that break things in production

Rotate evaluators — individual bias averages out

📊

LLM-as-Judge: Automating Evaluation

A controversial but increasingly common approach: use a strong model to evaluate a weaker one.

python

judge_prompt = """

You are an expert evaluator. Rate the following response on a scale

of 1-5 for each criterion.

User prompt: {prompt}

Model response: {response}

Criteria:

1. Accuracy (1-5): Are all facts correct?
2. Completeness (1-5): Does it fully address the question?
3. Clarity (1-5): Is it well-written and easy to understand?
4. Safety (1-5): Does it avoid harmful content?

For each criterion, provide a score and one-sentence justification.

"""

GPT-4 evaluates another model's output

evaluation = gpt4.generate(judge_prompt.format(
prompt=test_prompt,
    response=candidate_response

))

Advantages:

Cheap and fast (vs hiring human evaluators)

Consistent (same rubric every time)
Scalable (evaluate thousands of outputs)

Disadvantages:

Judge model has its own biases (prefers verbose, confident responses)

Can't evaluate things it's bad at itself

Circular logic risk: using GPT-4 to evaluate models trained to mimic GPT-4

The hybrid approach: Use LLM-as-judge for broad screening, human evaluators for final decisions on close calls.

Building Your Own Eval Suite

For production AI, public benchmarks are a starting point. Your custom eval suite is what actually matters.

python

Your eval suite should include:

1. Domain-specific accuracy

domain_tests = load_test_cases("eval/domain_knowledge.jsonl")

200+ Q&A pairs from YOUR domain, with verified answers

2. Regression tests

regression_tests = load_test_cases("eval/regression.jsonl")

Cases where previous models failed — make sure new model doesn't too

3. Safety tests

safety_tests = load_test_cases("eval/safety.jsonl")

Prompt injection attempts, harmful requests, sensitive topics

4. Format compliance

format_tests = load_test_cases("eval/format.jsonl")

Does it output valid JSON when asked? Follow instructions precisely?

5. Latency benchmarks

latency_tests = load_test_cases("eval/latency.jsonl")

P50, P95, P99 response times under load

6. Cost analysis

cost_tests = load_test_cases("eval/cost.jsonl")

Average tokens per response — is the new model more verbose (expensive)?

def run_full_eval(model):

results = {}
    results['domain_accuracy'] = evaluate(model, domain_tests)
    results['regression_pass_rate'] = evaluate(model, regression_tests)
    results['safety_score'] = evaluate(model, safety_tests)
    results['format_compliance'] = evaluate(model, format_tests)
    results['latency_p95'] = benchmark_latency(model, latency_tests)
    results['avg_output_tokens'] = measure_verbosity(model, cost_tests)
    return results

The Honest Truth About Evaluation

After working with dozens of models in production, here's what I've learned:

No single metric captures model quality. Use a dashboard, not a number.

User feedback > benchmarks. If users are happy, the model is good. Full stop.

Eval is never "done." As your use case evolves, so should your eval suite.

The best eval is production monitoring. A/B tests on real traffic tell you more than any offline benchmark.

Vibes are data too. If the model "feels wrong" to your team, it probably is — even if the numbers look fine.

🔬

Practical Takeaways

MMLU and HumanEval are starting points, not conclusions — they're contaminated and don't reflect real use

LMSYS Arena is the most trustworthy public benchmark — human preferences on real prompts

Build your own eval suite — domain-specific tests, regression tests, safety tests, latency benchmarks

Use LLM-as-judge for scale, humans for final calls — the hybrid approach wins

Track eval metrics over time — model quality can drift, especially after updates

Benchmark skepticism is healthy — if a model claims SOTA on everything, dig into the methodology

🛡️

What's Next?

Episode 92: How ChatGPT Works — You've learned every component: tokens, transformers, inference, scaling, evaluation. Now let's put it all together. How does a single message go from your keyboard to OpenAI's servers, through load balancers and inference clusters, and back to your screen as streaming text? The full architecture, end to end.

← Previous

Ep 90: Synthetic Data

Next →

Ep 92: How ChatGPT Works

Next: Episode 92 — How ChatGPT Works

You type a question. Two seconds later, words stream onto your screen. Here's everything that happens in between.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.