Synthetic Data: When AI Trains on Its Own Output

We have a problem. A big one.

The internet has roughly 5-10 trillion tokens of high-quality English text. GPT-4 was trained on about 13 trillion tokens. Llama 3 used 15 trillion. We're running out of human-written data. And foundation models are data-hungry beasts that need MORE.

The solution the industry landed on: have AI generate its own training data.

It sounds insane. Like a snake eating its own tail. Like photocopying a photocopy a thousand times. And in some cases, it IS insane — it leads to model collapse, where models trained on AI-generated data progressively degrade until they produce gibberish.

But done carefully, synthetic data is one of the most powerful techniques in modern AI. Let's understand when it works, when it fails, and why it's the future whether we like it or not.

The Stanford Alpaca Moment

In March 2023, Stanford released Alpaca — a fine-tuned LLaMA 7B model that cost $600 to create and performed surprisingly close to GPT-3.5.

Their method was beautifully simple:

Take GPT-3.5 (text-davinci-003)

Give it 175 human-written instruction-response pairs as examples

Ask it to generate 52,000 MORE instruction-response pairs in the same style

Fine-tune LLaMA 7B on those 52K synthetic examples

python

The self-instruct prompt (simplified)

prompt = """

You are asked to generate diverse task instructions and responses.

Here are some examples:

###

Instruction: Give three tips for staying healthy.

Response: 1. Eat a balanced diet... 2. Exercise regularly... 3. Get enough sleep...

###

Now generate a new, different instruction and response:

"""

GPT-3.5 generates thousands of these

synthetic_pair = gpt35.complete(prompt)

Result: A $600 model that could follow instructions nearly as well as a model that cost millions to train.

This was the gunshot that started the synthetic data race.

How Synthetic Data Is Used Today

1. Self-Instruct: Generating Training Pairs

The Alpaca approach, refined:

python

Modern self-instruct pipeline

def generate_synthetic_training_data(

teacher_model,      # Strong model (GPT-4, Claude)
    seed_examples,      # ~200 human-written examples
    num_to_generate,    # Target: 50K-500K
    diversity_prompt     # Ensures variety

):

synthetic_data = []

for batch in range(num_to_generate // 10):
        # Sample random seed examples for diversity
        seeds = random.sample(seed_examples, 3)

prompt = f"""Generate 10 diverse instruction-response pairs.

They should cover different topics and difficulty levels.

Make them specific and detailed.

Examples for style reference:

{format_examples(seeds)}

Requirements:

• Each must be unique (no repetition)
• Mix of simple and complex tasks
• Include: coding, writing, analysis, math, creative
• Responses should be detailed (100+ words)

"""

pairs = teacher_model.generate(prompt)
        synthetic_data.extend(parse_pairs(pairs))

return synthetic_data

1. Distillation: Transferring Knowledge from Large to Small

Knowledge distillation uses a large "teacher" model to generate training data for a smaller "student" model:

Teacher (GPT-4, 1.8T params) → generates high-quality responses
↓
Student (Llama 8B) ← fine-tuned on teacher's responses

This is how most open-source instruction-tuned models are made. Phi-3, Orca, WizardLM — they all used synthetic data from GPT-4 or Claude.

The trick is generating data that captures the teacher's reasoning process:

python

Generate with chain-of-thought for better distillation

prompt = """

Solve this step by step, showing your complete reasoning:

Question: A train leaves Mumbai at 9 AM traveling at 80 km/h.

Another train leaves Delhi at 10 AM traveling at 100 km/h toward Mumbai.

If Mumbai-Delhi distance is 1400 km, when do they meet?

Think through each step carefully before giving the answer.

"""

The student learns the REASONING PATTERN, not just the answer

teacher_response = gpt4.generate(prompt)

"Step 1: In the first hour, only Train A is moving...

Step 2: After 10 AM, both trains are approaching...

Step 3: Combined speed is 180 km/h..."

1. Data Augmentation: Expanding Limited Datasets

When you have some real data but not enough:

python

Original: 500 customer support conversations

Need: 10,000 for fine-tuning

augmentation_prompt = """

Here's a real customer support conversation:

Customer: "My order hasn't arrived and it's been 2 weeks"

Agent: "I'm sorry to hear that. Let me look up your order. Could you provide your order number?"

Generate 5 variations of this conversation with:

• Different products/issues but same pattern
• Different customer tones (frustrated, polite, confused)
• Slightly different agent responses (same quality)
• Natural language variations (not robotic)

"""

1. Preference Data for RLHF

Generating preference pairs — "this response is better than that one" — is expensive with humans. AI can help:

python

Generate multiple responses

responses = [model.generate(prompt) for _ in range(4)]

Use a judge model to rank them

ranking_prompt = f"""

Rank these responses from best to worst for the given prompt.

Consider: accuracy, helpfulness, safety, clarity.

Prompt: {prompt}

Response A: {responses[0]}

Response B: {responses[1]}

Response C: {responses[2]}

Response D: {responses[3]}

Ranking (best to worst):

"""

This creates preference pairs for DPO/RLHF training

ranking = judge_model.generate(ranking_prompt)

🔧

The Model Collapse Problem

Here's where it gets scary. Research from multiple groups (Oxford, Cambridge, Rice University) has shown:

When models are recursively trained on AI-generated data, they progressively degrade.

The mechanism:

Generation 1: Model A trained on human data → Good output

Generation 2: Model B trained on Model A's output → Slightly worse

Generation 3: Model C trained on Model B's output → Noticeably worse

...

Generation N: Model trained on recursive AI output → Garbage

Why it happens:

Distribution narrowing. AI output is less diverse than human output. It gravitates toward "average" responses. With each generation, the distribution gets narrower — tail events (creative, unusual, surprising content) get progressively eliminated.

Error accumulation. Every model makes small errors. Training on those errors amplifies them. Like photocopying a photocopy — each generation gets blurrier.

Mode collapse. The model starts repeating the same patterns. If the synthetic data contains any biases or repeated structures, they get reinforced exponentially.

Human text distribution: ████████████████████████ (wide, diverse)

AI gen 1: ██████████████████ (slightly narrower)

AI gen 2: ██████████████ (narrower still)

AI gen 3: ██████████ (losing diversity)

...

AI gen N: ████ (collapsed, repetitive)

The "Habsburg AI" Problem

The internet is increasingly filled with AI-generated content. Future models will be trained on data that includes AI-generated text from previous models. This recursive training — without tracking what's human vs synthetic — is called the "Habsburg AI" problem (like royal inbreeding leading to genetic degradation).

Some estimates suggest 50%+ of internet text will be AI-generated by 2026. If we're not careful about data provenance, every model trained on web crawls will be partially training on previous models' outputs.

📊

When Synthetic Data Works (and Doesn't)

Scenario Why

❌ Fails:

Scenario Why

The Golden Rule:

Synthetic data to teach FORMAT and REASONING: ✅

Synthetic data to teach FACTS and KNOWLEDGE: ❌

Synthetic data mixed with real data: ✅

Synthetic data replacing real data entirely: ❌

Best Practices for Synthetic Data

python

1. Always mix synthetic with real data

training_data = real_data + synthetic_data
assert len(real_data) / len(training_data) >= 0.3  # At least 30% real

2. Filter synthetic data aggressively

def filter_synthetic(pair):

# Check for common failure modes
    if is_repetitive(pair['response']):
        return False
    if contains_ai_artifacts(pair['response']):  # "As an AI language model..."
        return False
    if len(pair['response']) < 50:
        return False
    if is_near_duplicate(pair, existing_data):
        return False
    return True

synthetic_data = [p for p in raw_synthetic if filter_synthetic(p)]

3. Verify factual claims

for pair in synthetic_data:

if contains_factual_claims(pair['response']):
        if not verify_against_knowledge_base(pair['response']):
            pair['needs_human_review'] = True

4. Track provenance

for pair in synthetic_data:

pair['metadata'] = {
        'source': 'synthetic',
        'generator': 'gpt-4-turbo',
        'generation': 1,  # Never train on gen 2+ synthetic
        'timestamp': datetime.now(),
        'seed_example_id': pair.get('seed_id')
    }

The Future: Synthetic All the Way Down

Despite the risks, synthetic data is the future because we have no alternative:

Data exhaustion: We're running out of human internet text

Privacy: Synthetic data doesn't contain PII

Cost: $600 (Alpaca) vs $100M+ (human annotation at scale)

Languages: Generating synthetic data in low-resource languages is easier than collecting it

Domains: Medical, legal, financial data is restricted. Synthetic versions can be shared freely

The companies that master synthetic data generation — filtering, verification, mixing ratios, provenance tracking — will have a massive advantage. It's not about whether to use synthetic data. It's about how to use it without poisoning your models.

🔬

Practical Takeaways

Synthetic data works for teaching style and reasoning, not facts — don't generate "knowledge"

Always mix with real data — minimum 30% real to keep the distribution grounded

Model collapse is real — never train on data generated by models trained on your synthetic data

Filter aggressively — AI-generated data has systematic biases (verbosity, hedging, repetition)

Track provenance — know which data is human-written vs synthetic, and which generation

Verify factual claims — synthetic data will hallucinate; catch it before training

🛡️

What's Next?

Episode 91: Evaluation Frameworks — You've trained your model on a mix of real and synthetic data. But how do you know if it's actually good? MMLU, HumanEval, LMSYS Arena, vibe eval — the world of AI benchmarks is messy, gameable, and deeply flawed. Let's figure out what we can actually trust.

← Previous

Ep 89: Scaling Laws

Next →

Ep 91: Evaluation Frameworks

Next: Episode 91 — Evaluation Frameworks

'89.3% on MMLU' sounds impressive but means almost nothing for production. AI evaluation is broken. Here's what actually works.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.