RLHF & DPO: How Models Learn to Be Helpful (Not Just Correct)

GPT-3 was brilliant. It could write essays, code, poetry. But it was also a nightmare. Ask it to help plan a birthday party and it might give you a recipe for methamphetamine. Ask for a joke and get a racist rant.

The model was smart. It just wasn't aligned.

GPT-3 knew the answer to your question — it also knew conspiracy theories, toxic content, and copyrighted material, and it would mix them all together with zero judgment. The raw pretrained model optimizes for one thing: predicting the next token. It doesn't have a concept of "helpful" or "harmful."

RLHF (Reinforcement Learning from Human Feedback) is what transformed GPT-3 into ChatGPT. It's the difference between a genius with no social skills and a genius you'd actually want to talk to.

The Three Stages of Making a Useful Model

Every ChatGPT-like model goes through this pipeline:

Stage 1: PRETRAINING

→ "Read the entire internet, learn language patterns"
         → Result: Smart but uncontrollable

Stage 2: SUPERVISED FINE-TUNING (SFT)

→ "Here are examples of good conversations, learn this format"
         → Result: Follows instructions, but quality is uneven

Stage 3: RLHF / DPO

→ "Between these two answers, THIS one is better. Learn why."
         → Result: Consistently helpful, harmless, honest

Stage 1 takes 6 months and $1 billion. Stage 2 takes a week and 10,000 examples. Stage 3 is where the magic happens.

RLHF: The Original Approach

Step 1: Collect Human Preferences

Show human annotators TWO responses to the same prompt and ask: "Which is better?"

Prompt: "How do I fix a leaking faucet?"

Response A: "A leaking faucet is typically caused by a worn washer.

Turn off the water supply, remove the handle with a screwdriver,

replace the rubber washer ($2 at any hardware store), reassemble.

If it's a cartridge faucet, you'll need to replace the entire

cartridge — check YouTube for your specific brand."

Response B: "You should call a plumber. Plumbing is complex and

attempting DIY repairs could cause water damage. Professional

plumbers are trained to handle these situations safely."

Human preference: A ✓ (more helpful, actionable)

This is done thousands of times. The resulting dataset looks like:

json

{

"prompt": "How do I fix a leaking faucet?",
    "chosen": "A leaking faucet is typically caused by...",
    "rejected": "You should call a plumber..."

}

Step 2: Train a Reward Model

The reward model learns to predict which response a human would prefer. It takes a prompt + response and outputs a scalar score:

python

Conceptual reward model

reward_model("How do I fix a faucet?", response_A) → 0.85
reward_model("How do I fix a faucet?", response_B) → 0.32

The reward model is trained on the preference data:

python

Training objective: reward(chosen) > reward(rejected)

loss = -log(sigmoid(reward(chosen) - reward(rejected)))

This is a standard classification problem. The reward model learns the PATTERNS of what humans prefer — helpful, specific, honest answers over vague, hedging, or harmful ones.

Step 3: Optimize the Model with PPO

Now you use the reward model to train the actual language model using PPO (Proximal Policy Optimization), a reinforcement learning algorithm:

1. Generate a response to a prompt
2. Score it with the reward model
3. If score is high → reinforce this behavior (increase probability)
4. If score is low → discourage this behavior (decrease probability)
5. Repeat thousands of times

The key constraint: KL divergence penalty. You don't want the model to change too much from the SFT model. Without this constraint, the model would learn to "hack" the reward model — finding degenerate responses that score high but are nonsensical.

python

PPO objective (simplified)

loss = -reward_score + beta * KL(policy_modelreference_model)

^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

maximize don't change too much from the original

reward

The RLHF Pipeline

Human Preferences

↓

Reward Model (trained on preferences)

↓

PPO Training Loop:

→ Generate response

→ Score with reward model

→ Update policy (with KL constraint)

→ Repeat

↓

Aligned Model

Why RLHF Is Hard

Expensive. You need thousands of human annotations ($10-50 per hour per annotator).

Unstable. PPO training is notoriously finicky — hyperparameters matter a LOT.

Three models in memory. During PPO, you need: the policy model, the reference model, and the reward model. For a 70B model, that's ~420 GB of VRAM.

Reward hacking. The model finds outputs that score high on the reward model but aren't actually good. Like a student who learns to ace the test without understanding the material.

🔧

DPO: The Simpler Alternative

Direct Preference Optimization (published 2023) asked: "What if we skip the reward model entirely?"

DPO's insight: You can derive a closed-form solution for the RLHF objective. Instead of training a reward model and then doing RL, you can directly optimize the language model on preference data.

The DPO Formula

Loss = -log(sigmoid(β * (log π(chosenprompt) / log π_ref(chosenprompt))

- (log π(rejected|prompt) / log π_ref(rejected|prompt))))

In English:

Increase the probability of chosen responses (relative to the reference model)

Decrease the probability of rejected responses (relative to the reference model)

β controls how much the model can deviate from the reference

DPO vs RLHF

RLHF DPO

DPO in Practice

python

from trl import DPOTrainer, DPOConfig

from transformers import AutoModelForCausalLM, AutoTokenizer

Load your SFT model (the model you want to align)

model = AutoModelForCausalLM.from_pretrained("./my-sft-model")
ref_model = AutoModelForCausalLM.from_pretrained("./my-sft-model")
tokenizer = AutoTokenizer.from_pretrained("./my-sft-model")

Prepare preference dataset

Format: {"prompt": "...", "chosen": "...", "rejected": "..."}

dataset = load_dataset("json", data_files="preferences.jsonl")

DPO config

dpo_config = DPOConfig(
output_dir="./dpo-aligned-model",
    beta=0.1,               # KL penalty strength
    learning_rate=5e-7,      # Very low LR for alignment
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    num_train_epochs=1,      # Usually just 1 epoch
    warmup_ratio=0.1,
    logging_steps=10,
    bf16=True,

)

Train

trainer = DPOTrainer(
model=model,
    ref_model=ref_model,
    args=dpo_config,
    train_dataset=dataset["train"],
    tokenizer=tokenizer,

)

trainer.train()

That's it. Standard fine-tuning with a special loss function. No reward model, no RL loop, no PPO hyperparameter nightmares.

📊

Preference Data: What Makes a Good Dataset

Sources of Preference Data

Human annotation — Most expensive, highest quality. Hire annotators to rank response pairs.

AI feedback — Use GPT-4 to judge which response is better. Cheaper but can introduce biases.

Implicit signals — User upvotes/downvotes, regeneration requests, conversation abandonment.

Existing datasets — Anthropic's HH-RLHF, UltraFeedback, OpenAssistant.

Creating Preference Pairs

python

Generate 2 responses at different temperatures

import openai

prompt = "Explain quantum computing to a teenager"

Response A: lower temperature (more focused)

response_a = generate(prompt, temperature=0.3)

Response B: higher temperature (more creative but risky)

response_b = generate(prompt, temperature=1.0)

Have GPT-4 judge (or humans)

judgment = openai.chat.completions.create(
model="gpt-4",
    messages=[{
        "role": "user",
        "content": f"""Which response is better for a teenager?

Prompt: {prompt}

Response A: {response_a}

Response B: {response_b}

Reply with just "A" or "B" and a brief reason."""

}]

)

How Much Preference Data?

Model Size Minimum Recommended Source

Constitutional AI: Rules Without Humans

Anthropic introduced Constitutional AI — a variant where the AI critiques its OWN responses based on a set of principles (the "constitution"):

Principle 1: "Choose the response that is most helpful"

Principle 2: "Choose the response that is least likely to cause harm"

Principle 3: "Choose the response that is most honest about its limitations"

The process:

Generate a response

Ask the model: "Does this response follow the principles? How could it be improved?"

Model revises its response

Use the (original, revised) pair as preference data

Train with DPO

No human annotation needed for the preference pairs — just for writing the constitution.

Beyond RLHF: What's Next

Method Year Key Idea

RLHF (PPO) 2022 Reward model + RL

DPO 2023 Direct optimization on preferences

KTO 2024 Only needs thumbs up/down (no pairs)

ORPO 2024 Combines SFT and alignment in one step

SimPO 2024 No reference model needed

The trend is clear: alignment is getting simpler and cheaper. What used to require a team of RL engineers now takes a single training script.

🔬

Practical Takeaways

RLHF transforms smart models into useful ones — raw pretrained models are powerful but uncontrollable

DPO is the practical choice for most teams — 95% of RLHF quality, 10% of the complexity

Preference data > reward engineering — invest in quality preference pairs, not clever reward functions

One epoch of DPO is usually enough — alignment is a light touch, not heavy training

Constitutional AI reduces annotation costs — let the model critique itself based on principles

The field is moving toward simpler methods — ORPO and SimPO are worth watching

🛡️

What's Next?

Episode 50: Prompt Injection — You've built a helpful, aligned model. Now someone types "Ignore all previous instructions and..." and your model complies. Prompt injection is the biggest unsolved security problem in AI, and it's fundamentally harder than you think.

← Previous

Ep 48: Model Merging

Next →

Ep 50: Prompt Injection

Next: Episode 50 — Prompt Injection

One sentence broke every AI chatbot on the internet. And there is no complete fix. Not yet. Maybe not ever.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.