How Midjourney Works: From Text to Stunning Images

You type "a cyberpunk street in Mumbai, neon lights, rain, 8k" and get back art that belongs in a gallery.

Not stock photos. Not filters. Original images conjured from noise and math.

Midjourney, DALL-E, Stable Diffusion — they all use diffusion models. But Midjourney's aesthetic is distinct. Dreamlike. Cinematic. Almost painterly.

How does it work? Let's trace the journey from text to pixels.

The Core Idea: Reverse Diffusion

Diffusion models learn to reverse a destruction process:

Forward Process (Training)

Start with a clear image

Add noise gradually over 1000 steps

Image becomes pure static

Reverse Process (Generation)

Start with pure noise

Predict what noise was added

Remove it slightly

Repeat 50-100 times

An image emerges

Like watching a sculptor remove marble to reveal the statue inside — except the statue was always there in the math.

The Architecture: Latent Diffusion

Midjourney doesn't work directly on pixels. It works in latent space.

The Pipeline

Text Prompt → CLIP Encoder → Text Embedding

↓

Random Noise ← U-Net ← Denoising Steps

↓

Latent Representation

↓

VAE Decoder → Final Image

Components Explained

Component Function

CLIP Encoder Converts text to numerical embeddings

U-Net Predicts noise to remove at each step

VAE Compresses/Decompresses between pixel and latent space

Scheduler Controls how much noise to remove each step

🔧

Step-by-Step Generation

Step 1: Text Encoding

Your prompt "a cyberpunk street in Mumbai" is converted to embeddings using CLIP:

python

import clip

model, preprocess = clip.load("ViT-B/32")

text = clip.tokenize(["a cyberpunk street in Mumbai"])
text_features = model.encode_text(text)

text_features shape: [1, 512] - numerical representation of meaning

These embeddings guide the entire generation process.

Step 2: Initialize Noise

python

import torch

Start with random Gaussian noise

latents = torch.randn(1, 4, 64, 64)  # Batch, channels, height, width

This noise is in latent space — 64x64 with 4 channels, not the final 1024x1024 image.

Step 3: Denoising Loop

python

for i, t in enumerate(scheduler.timesteps):

# Predict noise in the current latent
    with torch.no_grad():
        noise_pred = unet(latents, t, encoder_hidden_states=text_embeddings)

# Compute how much to denoise this step
    latents = scheduler.step(noise_pred, t, latents).prev_sample

# After 50-100 steps, latents contain image information

Each iteration removes a calculated amount of noise, guided by the text embeddings.

Step 4: Decode to Pixels

python

VAE decoder turns latent representation into actual image

with torch.no_grad():

image = vae.decode(latents / 0.18215).sample

Output: [1, 3, 512, 512] - RGB image

📊

The U-Net: Heart of the Process

The U-Net architecture predicts noise. It's called "U" because of its shape:

Input (noisy latent)
↓
Encoder (downsampling, extracting features)
↓
Bottleneck (compressed representation)
↓
Decoder (upsampling, reconstructing)
↓
Output (predicted noise)

Cross-Attention: Where Text Meets Image

The magic happens in cross-attention layers:

Image Features ──┐

├──→ Cross-Attention ──→ Text-Guided Features

Text Embedding ──┘

At these layers, the model "looks" at the text embedding and decides which parts of the image to emphasize.

Classifier-Free Guidance (CFG)

This is what makes images "follow the prompt" instead of being loosely related.

python

Two predictions:

noise_conditional = unet(latents, t, text_embeddings)      # With text
noise_unconditional = unet(latents, t, null_embeddings)    # Without text

Scale the difference

noise_pred = noise_unconditional + cfg_scale * (noise_conditional - noise_unconditional)

CFG Scale Result

Midjourney's default CFG (~7) strikes its signature balance.

Upscaling: From Good to Stunning

The base generation is 512x512 or 1024x1024. The upscaler makes it print-ready.

Two-Stage Upscaling

Stage 1: Latent Upscaling

Upscale in latent space (cheaper)

Apply more denoising steps

Adds detail without hallucination

Stage 2: Super-Resolution

ESRGAN or similar upscaler

Trained on high-quality image pairs

Adds fine textures, sharpness

Comparison

Resolution Base Upscaled

🔬

Aesthetic Training: The Midjourney "Look"

Why do Midjourney images look distinct from DALL-E or Stable Diffusion?

Fine-Tuning on Curated Data

After base training, Midjourney fine-tunes on:

High-rated user generations

Professional photography

Artistic compositions

Specific aesthetic preferences

Reinforcement Learning from Human Feedback (RLHF)

Generate 4 variations → Users vote for favorites →

Model learns what humans prefer → Repeat millions of times

This creates a feedback loop that pushes the aesthetic in a specific direction — cinematic, painterly, detailed.

🛡️

Prompt Engineering for Diffusion

Unlike LLMs, diffusion models respond to specific syntax:

Token Weighting

"a (cyberpunk:1.5) street" → Emphasize cyberpunk

"a (rain:0.5) scene" → De-emphasize rain

Multi-Prompts

"cyberpunk street::2 neon lights::1 rain::0.5"

Each part gets different weight in the final image.

Negative Prompts

Prompt: "beautiful landscape"

Negative: "blurry, low quality, oversaturated, watermark"

Tells the model what to avoid.

📦

Hardware Requirements

Running diffusion locally:

Model VRAM Required Speed (RTX 4090)

SD 1.5 4GB 10 images/min

SDXL 8GB 4 images/min

FLUX 24GB 1 image/min

Midjourney API N/A (cloud) ~1 min/image

Midjourney runs on enterprise-grade clusters. You can't self-host it.

🚀

Practical Takeaways

Diffusion = reverse noise removal — Start static, gradually reveal structure

Latent space is efficient — Work compressed, decode at the end

CLIP connects text to images — Text embeddings guide every denoising step

CFG controls prompt adherence — Higher = more literal, lower = more creative

Upscaling is separate — Base generation + super-resolution pipeline

Aesthetic is trained — RLHF from millions of user preferences creates the "look"

Prompt engineering matters — Token weights, multi-prompts, negatives

🔄

What's Next?

Episode 96: How Voice Assistants Work — STT → LLM → TTS pipeline, Whisper, wake words, and the latency challenges of real-time voice AI.

← Previous

Ep 94: How Perplexity Works

Next →

Ep 96: How Voice Assistants Work

Next: Episode 96 — How Voice Assistants Work

You speak. Two seconds later, a voice responds with the weather. Three AI systems chained together make this work.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.