Training Data for Fine-Tuning: Quality Is the Only Thing That Matters

You've got LoRA set up. QLoRA is humming on your RTX 4090. You fire up training and... the model outputs garbage. Worse than the base model. Three days of training, wasted.

The model isn't broken. Your data is.

Fine-tuning is 20% code and 80% data. The model architecture, the hyperparameters, the LoRA rank — those matter. But the training data determines everything. Bad data with perfect code produces a bad model. Great data with mediocre code produces a great model.

Let's talk about what great data looks like.

Data Formats: The Big Three

Every fine-tuning framework expects data in a specific format. There are three dominant ones:

1. Alpaca Format

The OG instruction-following format. Simple and clean:

json

{

"instruction": "Summarize the following text in 2 sentences",
    "input": "The transformer architecture was introduced in 2017...",
    "output": "The transformer architecture, introduced in 2017, replaced RNNs with self-attention mechanisms. It became the foundation for all modern language models including GPT and BERT."

}

Three fields:

instruction: What to do

input: Context (optional — can be empty string)

output: Expected response

Best for: Simple instruction-following tasks, single-turn interactions.

1. ShareGPT Format

Multi-turn conversation format. Named after the (now-defunct) site where people shared ChatGPT conversations:

json

{

"conversations": [
        {"from": "human", "value": "I need to parse JSON in Python but it might be malformed. What's the safest way?"},
        {"from": "gpt", "value": "Use a try-except block with json.loads():\n\n```python\nimport json\n\ntry:\n    data = json.loads(raw_string)\nexcept json.JSONDecodeError as e:\n    print(f'Invalid JSON: {e}')\n    data = None\n```\n\nFor extra safety, validate the schema after parsing with jsonschema or pydantic."},
        {"from": "human", "value": "What if the JSON has comments in it? Some configs use // comments"},
        {"from": "gpt", "value": "Standard json.loads() will fail on comments. Use json5 or strip comments first:\n\n```python\nimport json5  # pip install json5\ndata = json5.loads(raw_string)  # Handles comments and trailing commas\n```"}
    ]

}

Best for: Chat models, multi-turn interactions, conversational assistants.

1. Chat Template Format (Messages)

The modern standard, aligned with OpenAI's message format:

json

{

"messages": [
        {"role": "system", "content": "You are a senior Python developer who gives concise, production-ready code."},
        {"role": "user", "content": "Write a retry decorator with exponential backoff"},
        {"role": "assistant", "content": "```python\nimport time\nfrom functools import wraps\n\ndef retry(max_retries=3, base_delay=1):\n    def decorator(func):\n        @wraps(func)\n        def wrapper(*args, **kwargs):\n            for attempt in range(max_retries):\n                try:\n                    return func(*args, **kwargs)\n                except Exception as e:\n                    if attempt == max_retries - 1:\n                        raise\n                    time.sleep(base_delay * 2 ** attempt)\n        return wrapper\n    return decorator\n```"}
    ]

}

Best for: Most modern fine-tuning. Works with virtually every framework.

Which Format to Use?

Format Multi-turn System prompt Framework support Recommendation

Alpaca No No Universal Legacy, but simple

Use the messages/chat template format unless you have a specific reason not to.

Quality Over Quantity: The Numbers

Here's data that will change how you think about fine-tuning:

Dataset Size Quality Result

The LIMA paper (Less Is More for Alignment) showed that just 1,000 carefully curated examples can produce a model that rivals models trained on 50,000+ examples.

Why? Because fine-tuning isn't teaching the model new knowledge (that happened during pretraining). Fine-tuning is teaching the model a behavior pattern. And you can demonstrate a pattern in 500-1,000 examples.

The Quality Checklist

Every single example in your dataset should pass these checks:

Is the output what an expert would write? If not, remove it.

Is the instruction clear and unambiguous? Vague instructions teach vague behavior.

Is the output the RIGHT length? If you want concise answers, your training data should have concise answers.

Does it handle edge cases? Include examples where the right answer is "I don't know" or "I need more information."

Is it diverse? 500 examples of the same pattern teaches one trick. 500 diverse examples teach general behavior.

🔧

Data Cleaning: The Boring Part That Matters Most

Raw data is always dirty. Here's what to look for:

Common Problems

python

Problem 1: Duplicate or near-duplicate examples

Tool: deduplicate with MinHash or exact string matching

from datasketch import MinHash, MinHashLSH

Problem 2: Inconsistent formatting

Some outputs use markdown, others plain text, others HTML

Solution: Standardize everything to one format

Problem 3: Wrong or outdated information

GPT-3.5 generated data often has subtle errors

Solution: Human review of at least a random 10% sample

Problem 4: Truncated responses

Outputs that end mid-sentence because of token limits

Solution: Filter by checking for proper sentence endings

Problem 5: System prompt leakage

Training data that accidentally includes "As an AI language model..."

Solution: Regex filter for common AI hedging phrases

Automated Cleaning Pipeline

python

import json

import re

def clean_example(example):

"""Clean a single training example. Returns None if should be removed."""

messages = example.get("messages", [])

# Remove if too short
    assistant_msgs = [m for m in messages if m["role"] == "assistant"]
    if any(len(m["content"]) < 50 for m in assistant_msgs):
        return None

# Remove if contains AI hedging
    hedging_patterns = [
        r"as an ai",
        r"i don't have personal",
        r"i cannot provide",
        r"i'm just a language model",
    ]
    for msg in assistant_msgs:
        for pattern in hedging_patterns:
            if re.search(pattern, msg["content"], re.IGNORECASE):
                return None

# Remove if truncated (doesn't end with punctuation or code block)
    last_content = assistant_msgs[-1]["content"].strip()
    if not re.search(r'[.!?\n`]$', last_content):
        return None

# Normalize whitespace
    for msg in messages:
        msg["content"] = re.sub(r'\n{3,}', '\n\n', msg["content"]).strip()

return example

Process dataset

with open("raw_data.jsonl") as f:

raw = [json.loads(line) for line in f]

cleaned = [clean_example(ex) for ex in raw]
cleaned = [ex for ex in cleaned if ex is not None]
print(f"Kept {len(cleaned)}/{len(raw)} examples ({len(cleaned)/len(raw)*100:.1f}%)")

A typical cleaning pipeline removes 20-40% of raw data. That's normal. That's good.

📊

Synthetic Data: When You Don't Have Enough

You need 1,000 expert-quality examples but only have 50. Enter synthetic data generation — using a stronger model to generate training data for a weaker model.

The Process

1. Write 50 seed examples by hand (highest quality possible)
2. Use GPT-4 / Claude to generate variations
3. Human-review the generated examples
4. Fine-tune your target model on the reviewed set

Generating with GPT-4

python

import openai

SEED_EXAMPLE = """

User: Our API is returning 503 errors intermittently under load

Assistant: Intermittent 503s under load point to a few common causes:

1. Backend saturation — Your upstream servers are running out of worker threads/processes. Check your server's active connections vs max connections.

1. Load balancer health checks — If health checks are too aggressive, the LB might be marking healthy backends as down. Check your health check interval and threshold.

1. Connection pooling — Database connection exhaustion is the #1 hidden cause. Run SELECT count(*) FROM pg_stat_activity; to check.

Quick diagnosis: Run curl -v during a 503 and check the Server header. If it's from your load balancer vs your app server, that tells you where the bottleneck is.

"""

prompt = f"""Generate a similar technical support conversation.

The user should describe a different infrastructure problem.

The assistant should give the same depth and style of response.

Example:

{SEED_EXAMPLE}

Now generate a new, different example:"""

response = openai.chat.completions.create(
model="gpt-4",
    messages=[{"role": "user", "content": prompt}],
    n=5,  # Generate 5 variations
    temperature=0.8,

)

Rules for Synthetic Data

Always review generated data. GPT-4 makes mistakes. Every example needs at least a skim.

Use diverse prompts. Don't generate 1,000 examples from the same template — vary the generation prompt.

Mix synthetic and real. A 70% synthetic / 30% real mix often performs best.

Use the strongest model available. Synthetic data from GPT-4 is better than from GPT-3.5.

Evol-Instruct — Generate progressively harder versions of each example. Start simple, make it complex.

How Much Data Do You Need?

Task Minimum Sweet Spot Overkill

These numbers assume high-quality, curated data. If your data is noisy, multiply by 3-5×.

The scaling law: Doubling data quality has 5-10× the impact of doubling data quantity.

Dataset Structure for Training

Here's a production-ready dataset structure:

my_finetune_data/

├── train.jsonl # 90% of data

├── eval.jsonl # 10% of data (held out)

├── metadata.json # Dataset info

└── README.md # What this dataset is, how it was created

json

// metadata.json

{

"name": "customer-support-v2",
    "format": "messages",
    "total_examples": 2847,
    "train_examples": 2562,
    "eval_examples": 285,
    "created": "2024-12-15",
    "source": "70% real support tickets (anonymized), 30% GPT-4 synthetic",
    "cleaning": "Removed duplicates, truncated responses, PII",
    "avg_tokens_per_example": 650,
    "total_tokens": ~1.85M

}

Always have an eval split. Without it, you can't detect overfitting.

🔬

Practical Takeaways

Quality beats quantity, always — 500 perfect examples > 50,000 messy ones

Use the messages/chat template format — it's the modern standard

Clean aggressively — removing 30% of your data often improves results

Synthetic data works — but always human-review it

Hold out 10% for evaluation — you need to measure, not guess

Document your dataset — future you will thank present you

🛡️

What's Next?

Episode 47: Fine-Tune Evaluation — You've trained the model. But is it actually good? Perplexity, benchmarks, vibe checks, loss curves — how do you know if your fine-tune worked? And how do you catch overfitting before it ruins your model?

← Previous

Ep 45: QLoRA

Next →

Ep 47: Fine-Tune Evaluation

Next: Episode 47 — Fine-Tune Evaluation

Loss went down. Training finished. The response looks... okay? But 'okay' isn't a metric. Here's how to actually evaluate.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.