Diffusion Models: How AI Turns Noise Into Art

Take a photo. Now add a tiny bit of static noise. Then a bit more. More. Keep going until there's nothing left but pure random noise — TV static.

Now reverse it. Start from the noise and remove it step by step until a photo appears.

That's diffusion. That's literally how Stable Diffusion, DALL-E 3, and Midjourney create images. They learn to reverse noise — to look at random static and gradually sculpt it into a coherent image. It's like watching a sculptor chip away marble, except the sculptor is a neural network and the marble is Gaussian noise.

The Core Idea: Forward and Reverse

Diffusion models have two processes:

Forward Process (Destroying)

Take a real image and gradually add Gaussian noise over T timesteps until it becomes pure noise:

Step 0: [Clear photo of a cat]

Step 100: [Slightly noisy cat]

Step 500: [Very noisy, barely visible cat]

Step 1000: [Pure random noise — no cat visible]

Mathematically, at each step:

x_t = √(α_t)  x_{t-1} + √(1 - α_t)  ε

where:

x_t = image at timestep t
α_t = noise schedule (how much signal to keep)
ε = random Gaussian noise

This is easy — just keep adding noise. A child could do it.

Reverse Process (Creating)

Now the hard part: learn to go backwards. Given a noisy image at step t, predict what it looked like at step t-1:

Step 1000: [Pure random noise]

Step 900: [Vaguely shaped blobs]

Step 500: [Recognizable shapes emerge]

Step 100: [Almost clear image]

Step 0: [Clean, sharp photo of a cat]

The neural network learns this reverse step. It's trained on millions of image-noise pairs: "given this noisy image at step 500, predict the noise that was added." Once you can predict the noise, you can subtract it.

The Training Process

Training a diffusion model is surprisingly straightforward:

python

Simplified diffusion training loop

for image in dataset:

# 1. Pick a random timestep
    t = random.randint(0, 1000)

# 2. Add noise corresponding to that timestep
    noise = torch.randn_like(image)
    noisy_image = add_noise(image, noise, t)

# 3. Ask the model to predict the noise
    predicted_noise = model(noisy_image, t)

# 4. Loss = how wrong was the noise prediction?
    loss = MSE(predicted_noise, noise)

# 5. Update model weights
    loss.backward()
    optimizer.step()

The model doesn't learn to generate images directly. It learns to predict noise. During generation, you start with pure noise and iteratively subtract the predicted noise, step by step. Each step reveals a slightly clearer image.

🔧

U-Net: The Workhorse Architecture

The neural network that predicts noise is typically a U-Net — an architecture shaped like the letter U:

Input (noisy image)
↓

[Encoder] ─── downsample ───→ [Bottleneck] ───→ [Decoder] ─── upsample

64×64                      8×8                           64×64
    ↓                          ↓                             ↓
    128 channels              512 channels                 128 channels

└──── Skip connections ────────────────────────────────┘

Why U-Net?

Encoder — Progressively shrinks the image while increasing channels (captures what's in the image)

Bottleneck — Compressed representation (captures global structure)

Decoder — Progressively grows back to full resolution (reconstructs details)

Skip connections — Connect encoder layers directly to decoder layers (preserves fine details that would otherwise be lost in compression)

The U-Net also receives the timestep as input (telling it how noisy the image is) and the text embedding (telling it what to generate).

Timestep Conditioning

The model needs to know "how noisy is this image?" because denoising at step 900 (very noisy) requires a different strategy than step 100 (almost clean):

python

Timestep embedding (sinusoidal, like positional encoding)

t_embedding = sinusoidal_embed(timestep)  # [d_model]

Injected into every layer of the U-Net via addition or FiLM conditioning

📊

Text Conditioning: How Prompts Guide Generation

When you type "a cat wearing a top hat on the moon," how does the model know what to generate?

CLIP Text Encoder

The text prompt gets encoded by a CLIP text encoder (or T5 for newer models) into a sequence of embedding vectors:

"a cat wearing a top hat on the moon"

↓ CLIP Text Encoder

[embed_1, embed_2, ..., embed_N] (77 tokens × 768 dims for SD 1.5)

Cross-Attention

These text embeddings are injected into the U-Net via cross-attention layers. At multiple points in the U-Net, the image features attend to the text features:

Image features (queries) → attend to → Text embeddings (keys, values)

"Where should I put the cat?" → attention to "cat" token → cat-shaped denoising

"What should be on its head?" → attention to "top hat" token → hat-shaped denoising

This is the same cross-attention mechanism from transformers. The image learns to "look at" the relevant words to guide each pixel's denoising.

Classifier-Free Guidance (CFG)

Here's a critical trick that makes or breaks image quality. During generation, the model runs two predictions at each step:

Conditional prediction — "denoise this with the text prompt"

Unconditional prediction — "denoise this with no text prompt (empty string)"

Then the final prediction amplifies the difference:

final = unconditional + guidance_scale × (conditional - unconditional)
guidance_scale = 7.5 (typical)

What this does: It asks "what's the difference between generic denoising and prompt-guided denoising?" and then amplifies that difference. Higher guidance scale = the image follows the prompt more closely but can become oversaturated or artifact-y.

CFG Scale Effect

This is why you see a "CFG scale" slider in Stable Diffusion UIs — it controls how much the image listens to your prompt.

Latent Diffusion: The Speed Trick

Doing diffusion in pixel space (512×512×3 = 786,432 values) is insanely expensive. Latent Diffusion Models (LDM) — the breakthrough behind Stable Diffusion — do diffusion in a compressed latent space instead.

The VAE

A Variational Autoencoder (VAE) compresses images before diffusion and decompresses after:

Image (512×512×3) → VAE Encoder → Latent (64×64×4) → Diffusion → VAE Decoder → Image

Compression: 512×512×3 = 786,432 values

Latent: 64×64×4 = 16,384 values

Compression ratio: 48x fewer values!

The diffusion process operates entirely in this compressed latent space. 48x fewer values means 48x less compute per denoising step. This is what made Stable Diffusion runnable on consumer GPUs.

The Full Pipeline

Text prompt: "a cat wearing a top hat on the moon"

↓

CLIP Text Encoder → text embeddings

↓

Random latent noise (64×64×4)

↓

U-Net denoises in latent space (50 steps, guided by text embeddings)

↓

Clean latent (64×64×4)

↓

VAE Decoder → final image (512×512×3)

🔬

Denoising Steps: Quality vs Speed

More denoising steps = better quality but slower generation:

Steps Time (A100) Quality Use Case

Schedulers

The scheduler (also called sampler) controls the noise schedule — how much noise to remove at each step. Different schedulers have different tradeoffs:

Scheduler Steps Needed Quality Speed

DDPM 1000 Excellent Very slow

DDIM 50 Good Moderate

Euler 20-30 Good Fast

DPM++ 2M Karras 20-30 Excellent Fast

LCM 4-8 Good Very fast

DPM++ 2M Karras is the community favorite — excellent quality in 20-30 steps.

🛡️

SDXL and Beyond

Stable Diffusion XL (SDXL) improved on SD 1.5 with:

Feature SD 1.5 SDXL

Flux and SD3

The latest generation replaces the U-Net with a DiT (Diffusion Transformer) — using a transformer architecture instead of the traditional U-Net:

SD 1.5/SDXL: Text → U-Net (conv + attention) → Image

Flux/SD3: Text → DiT (pure transformer) → Image

DiTs scale better with compute and produce higher quality images. Flux by Black Forest Labs (the team behind Stable Diffusion) is currently considered the best open-source image model.

📦

ControlNet: Guided Generation

ControlNet adds structural guidance to diffusion models. Instead of just text, you can provide:

Control Type Input Use Case

Canny edges Edge map Preserve shapes

Text: "a photo of a woman in a red dress"

+ Pose control: [skeleton with specific arm positions]

= Generated image matching both the text AND the pose

ControlNet works by creating a trainable copy of the U-Net encoder that processes the control signal and adds its output to the main U-Net via addition. This preserves the original model's quality while adding structural control.

🚀

Image-to-Image: Starting From Something

Instead of starting from pure noise, you can start from a real image with noise added:

Start from noise: step 1000 → step 0 (full creative freedom)

Start from image: step 500 → step 0 (preserves original structure)

The denoising strength controls how much of the original image to keep:

Strength Noise Level Result

This is how "AI image editing" works — you start from your image, add the right amount of noise, and denoise with a new prompt.

🔄

Practical Takeaways

Diffusion = learn to reverse noise — add noise step by step, then train a model to remove it

U-Net predicts noise, not images — the model outputs what noise to subtract at each step

CFG controls prompt adherence — higher values = more prompt-following but potentially less natural

Latent diffusion is the speed trick — 48x compression via VAE makes consumer GPU generation possible

20-30 steps with DPM++ 2M Karras — sweet spot for quality vs speed

The future is DiT — transformers replacing U-Nets for better scaling

🏗️

What's Next?

Episode 61: State Space Models — Mamba is a challenger to the transformer throne. Linear scaling instead of quadratic. No attention mechanism at all. How does it work, and can it actually replace transformers?

← Previous

Ep 59: Voice & TTS

Next →

Ep 61: State Space Models

Next: Episode 61 — State Space Models

Self-attention scales quadratically with sequence length. Double the context, quadruple the compute. Mamba says there's a better way.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.