Fine-Tune Evaluation: How to Know If Your Model Is Actually Good

You trained the model. Loss went down. Training finished without errors. You open the chat interface, type a prompt, and the response looks... okay?

But "looks okay" isn't engineering. That's vibes. And vibes don't survive production.

The dirty secret of fine-tuning: most people have no idea if their model improved. They see the loss curve going down and ship it. Then users complain. Then they retrain with more data and ship again. Still no measurement.

Let's fix that. Here's how to actually evaluate a fine-tuned model.

The Loss Curve: Your First Signal

During training, two numbers matter:

Training Loss

How well the model predicts the next token on data it's SEEN. This always goes down. If it doesn't, something is broken.

Validation Loss

How well the model predicts the next token on data it HASN'T seen. This is the one that matters.

Epoch 1: Train Loss: 2.45Val Loss: 2.50 Epoch 2: Train Loss: 1.82Val Loss: 1.90 Epoch 3: Train Loss: 1.31Val Loss: 1.45 ← Sweet spot Epoch 4: Train Loss: 0.95Val Loss: 1.52 ← Overfitting starts Epoch 5: Train Loss: 0.61Val Loss: 1.78 ← Definitely overfitting Epoch 6: Train Loss: 0.34Val Loss: 2.10 ← Model is memorizing

The pattern:

Training loss keeps dropping — model is memorizing examples

Validation loss starts rising — model stopped learning generalizable patterns

The gap between them — the wider it gets, the more the model is memorizing

Reading the Loss Curve

Pattern Meaning Action

Both losses decreasing Model is learning Keep training

Always save checkpoints at every epoch. When overfitting starts, you want to go back to the best checkpoint.

Perplexity: The Standard Metric

Perplexity is the most common metric for language models. It answers: "How surprised is the model by the correct answer?"

Perplexity = e^(loss)
Loss = 1.5  →  Perplexity = e^1.5 = 4.48
Loss = 1.0  →  Perplexity = e^1.0 = 2.72
Loss = 0.5  →  Perplexity = e^0.5 = 1.65

Lower perplexity = model is less surprised = model is better at predicting.

Calculating Perplexity

python

import torch

import math

from transformers import AutoModelForCausalLM, AutoTokenizer

def calculate_perplexity(model, tokenizer, texts, max_length=512):
model.eval()
    total_loss = 0
    total_tokens = 0
for text in texts:
        inputs = tokenizer(text, return_tensors="pt",
                          truncation=True, max_length=max_length)
        inputs = {k: v.to(model.device) for k, v in inputs.items()}
with torch.no_grad():
            outputs = model(**inputs, labels=inputs["input_ids"])
            total_loss += outputs.loss.item() * inputs["input_ids"].size(1)
            total_tokens += inputs["input_ids"].size(1)
avg_loss = total_loss / total_tokens
    perplexity = math.exp(avg_loss)
    return perplexity

Compare base vs fine-tuned

base_ppl = calculate_perplexity(base_model, tokenizer, eval_texts)
ft_ppl = calculate_perplexity(finetuned_model, tokenizer, eval_texts)
print(f"Base perplexity: {base_ppl:.2f}")
print(f"Fine-tuned perplexity: {ft_ppl:.2f}")

Perplexity Gotchas

Only compare on the same eval set. Perplexity of 3.0 on your data vs 2.5 on someone else's data means nothing.

Perplexity on training data is meaningless. The model memorized it. Always use held-out data.

Low perplexity ≠ good model. A model that memorizes your eval set has low perplexity but is useless. Perplexity is necessary but not sufficient.

🔧

Benchmarks: Standardized Testing

Benchmarks test specific capabilities. The main ones:

Benchmark Tests Format Good Score

MMLU General knowledge (57 subjects) Multiple choice >60%

HumanEval Code generation Pass@1 >50%

GSM8K Math reasoning Open answer >60%

Running Benchmarks with lm-eval

bash

Install

pip install lm-eval

Run MMLU on your fine-tuned model

lm_eval --model hf \

--model_args pretrained=./my-finetuned-model \
    --tasks mmlu \
    --batch_size 4 \
    --output_path ./eval_results/

Run multiple benchmarks

lm_eval --model hf \

--model_args pretrained=./my-finetuned-model \
    --tasks mmlu,hellaswag,truthfulqa_mc2,gsm8k \
    --batch_size 4 \
    --output_path ./eval_results/

When Benchmarks Lie

Fine-tuning can improve benchmark scores without improving real-world quality. This happens when:

Your training data is similar to benchmark test data (contamination)

You fine-tune specifically to handle multiple-choice format

The benchmark doesn't test what your users actually need

Always pair benchmarks with task-specific evaluation.

📊

The Vibe Check: Don't Skip This

I'm serious. After all the metrics, sit down and actually talk to your model. The vibe check catches things metrics miss.

The Vibe Check Protocol

Create a test suite of 20-30 prompts that represent your actual use case:

markdown

Vibe Check Suite (Customer Support Bot)

1. Simple question with clear answer
2. Question that requires looking up policy
3. Angry customer complaint
4. Request that's outside our scope
5. Multi-part question
6. Follow-up question referencing previous context
7. Question with ambiguous intent
8. Edge case: empty or nonsensical input
9. Request for personal information
10. Comparison question (your product vs competitor)

Run each prompt through:

Base model (no fine-tuning)

Base model + good prompt (system prompt engineering)

Fine-tuned model

Score each response on:

Relevance (1-5): Does it answer the question?
Accuracy (1-5): Is the information correct?
Style (1-5): Does it match the desired tone?
Completeness (1-5): Is anything missing?

A/B Comparison

python

Simple A/B comparison framework

import random

def blind_comparison(prompt, model_a_response, model_b_response):

"""Randomly order responses so you evaluate without bias."""
    responses = [
        ("A", model_a_response),
        ("B", model_b_response)
    ]
    random.shuffle(responses)

print(f"Prompt: {prompt}\n")
    print(f"Response 1 ({responses[0][0]}):\n{responses[0][1]}\n")
    print(f"Response 2 ({responses[1][0]}):\n{responses[1][1]}\n")

winner = input("Which is better? (1/2/tie): ")
    return responses[int(winner)-1][0] if winner != "tie" else "tie"

Blind evaluation is critical. If you know which response is from your fine-tuned model, you'll unconsciously favor it.

Detecting Overfitting

Overfitting is the biggest risk in fine-tuning. Signs:

1. Memorized Responses

The model outputs training examples verbatim. Ask it something slightly different from a training example and it either repeats the training answer or falls apart.

Training example:  "What's our return policy?" → "30-day money-back guarantee..."

Test: "How do returns work?" → Completely different quality answer

If the model only handles exact phrasings from training data, it's overfit.

1. Loss Divergence

python

Check if validation loss is diverging

val_losses = [2.50, 1.90, 1.45, 1.52, 1.78, 2.10]
train_losses = [2.45, 1.82, 1.31, 0.95, 0.61, 0.34]

Overfitting ratio

for i in range(len(val_losses)):

ratio = val_losses[i] / train_losses[i]
    print(f"Epoch {i+1}: ratio = {ratio:.2f}")
    # ratio > 2.0 = likely overfitting

Epoch 1: ratio = 1.02 ← Normal

Epoch 3: ratio = 1.11 ← Still fine

Epoch 5: ratio = 2.92 ← Overfitting

Epoch 6: ratio = 6.18 ← Very overfit

1. Decreased Diversity

Ask the same question 5 times with temperature=0.7. A healthy model gives varied responses. An overfit model gives nearly identical ones — it's stuck in a narrow output distribution.

1. Catastrophic Forgetting

The model gets great at your task but loses general abilities. Test with general questions:

"What is the capital of France?"

"Write a haiku about rain."

"Explain photosynthesis to a 5-year-old."

If these degrade significantly, your model has forgotten too much. Solutions: less training, lower learning rate, or mix in some general data.

Human Evaluation: The Gold Standard

For production models, nothing replaces human evaluation:

Setting Up Human Eval

Create an eval set — 100-200 examples covering your use case

Get multiple evaluators — at least 3 people rating each example

Use a rubric — specific criteria, not just "good/bad"

Calculate inter-annotator agreement — if evaluators disagree a lot, your rubric needs work

Rubric Example

markdown

Rating: Helpfulness (1-5)

5 - Directly answers the question with actionable specifics

4 - Answers the question well, minor details missing

3 - Partially answers, needs follow-up

2 - Tangentially related but doesn't address the question

1 - Irrelevant, wrong, or harmful

LLM-as-Judge (Budget Alternative)

Can't afford human evaluators? Use GPT-4 as a judge:

python

judge_prompt = """Rate the following response on a scale of 1-5 for:
Relevance (does it answer the question?)Accuracy (is the information correct?)Helpfulness (would a user be satisfied?)

Question: {question}

Response: {response}

Return JSON: {"relevance": X, "accuracy": X, "helpfulness": X, "reasoning": "..."}"""

LLM-as-judge correlates ~80% with human judgment. Not perfect, but much better than nothing.

🔬

The Evaluation Playbook

Here's the order I evaluate every fine-tune:

1. Loss curves → Is the model learning? Is it overfitting?
2. Perplexity → Compare base vs fine-tuned on eval set
3. Vibe check → 20-30 manual tests, blind comparison
4. Benchmarks → MMLU, HumanEval, etc. (check for regression)
5. Task-specific eval → Your actual use case metrics
6. Human eval → 100+ examples rated by multiple people
7. A/B test in prod → The ultimate test

Don't skip steps. Don't ship after step 1.

🛡️

Practical Takeaways

Validation loss is your first signal — watch for the train/val divergence

Perplexity is necessary but not sufficient — low perplexity doesn't mean good model

The vibe check is not optional — sit down and actually talk to your model

Blind evaluation removes bias — randomize which model's output you're reading

Overfitting is the #1 fine-tuning failure mode — save checkpoints, monitor validation loss

Human eval is the gold standard — LLM-as-judge is a good budget alternative

📦

What's Next?

Episode 48: Model Merging — What if you could combine two fine-tuned models into one that has BOTH their abilities? Model merging is the wildest technique in the open-source AI community. TIES, DARE, linear merge, frankenmerge — it sounds crazy, and it works.

← Previous

Ep 46: Training Data for Fine-Tuning

Next →

Ep 48: Model Merging

Next: Episode 48 — Model Merging

One model is amazing at coding. Another at creative writing. You want both. Model merging lets you combine them — no training required.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.