Multi-modal AI: How Vision Models Actually See Your Images

When you upload a screenshot to ChatGPT and it reads the text, identifies the UI elements, and tells you the bug — what's actually happening?

It's not OCR. It's not a separate image analysis pipeline. The model is literally seeing your image as tokens — the same way it sees text. Your screenshot gets chopped into patches, converted into embeddings, and fed into the same transformer that handles language.

Text, images, audio — to a multi-modal model, they're all just sequences of vectors. The unification is beautiful. And understanding it changes how you use these tools.

What "Multi-modal" Actually Means

A modal is a type of input or output. Text is one modality. Images are another. Audio, video, 3D — each is a modality.

Modality Examples How AI Processes It

A multi-modal model handles two or more of these in a single architecture. GPT-4V handles text + images. Gemini handles text + images + audio + video. The key insight: they all get converted to the same representation — embedding vectors — before the transformer sees them.

How Images Become Tokens

This is the core trick. A transformer only understands sequences of vectors. So how do you feed it an image?

Step 1: Patch Extraction

The image gets divided into a grid of square patches. Think of it like cutting a photo into tiles:

Original image: 1024 × 1024 pixels

Patch size: 16 × 16 pixels

Grid: 64 × 64 = 4,096 patches

Each patch = one "image token"

A single 1024×1024 image becomes 4,096 tokens. That's why image inputs are expensive — they eat a huge chunk of your context window.

Step 2: Patch Embedding

Each 16×16 pixel patch (which is 16 × 16 × 3 = 768 raw values for RGB) gets projected into the model's embedding dimension through a linear layer or small CNN:

python

Simplified patch embedding

patch = image[y:y+16, x:x+16]  # 16×16×3 = 768 values
patch_flat = patch.flatten()     # [768]
embedding = linear(patch_flat)   # [768] → [d_model]

Now it's the same shape as a text token embedding!

Step 3: Position Encoding

Just like text tokens need position information, image patches need to know where they are in the grid. 2D position embeddings encode both row and column:

Patch (0,0) → position embedding for top-left
Patch (0,1) → position embedding for top-center

...

Patch (63,63) → position embedding for bottom-right

Step 4: Into the Transformer

Now you have a sequence of 4,096 vectors, each the same dimension as text token embeddings. The transformer processes them with standard self-attention — each patch attends to every other patch, learning spatial relationships.

The final output? A set of rich image representations that understand edges, objects, text, layout — everything.

🔧

CLIP: The Bridge Between Text and Images

CLIP (Contrastive Language-Image Pre-training) by OpenAI (2021) was the breakthrough that made modern multi-modal AI possible.

The Training Trick

CLIP trains two encoders simultaneously:

Image encoder — turns images into embedding vectors

Text encoder — turns text descriptions into embedding vectors

The training objective: make matching image-text pairs have similar embeddings and non-matching pairs have different embeddings.

Training pair 1: [Photo of dog on beach] ↔ "a dog playing on a sandy beach"

Training pair 2: [Photo of city skyline] ↔ "urban skyline at sunset"

Goal: embed(dog_photo) ≈ embed("dog on beach")

embed(dog_photo) ≠ embed("city skyline")

This was trained on 400 million image-text pairs scraped from the internet. The result: a model that understands the relationship between visual concepts and language.

Why CLIP Matters

CLIP creates a shared embedding space where text and images coexist. You can:

Search images with text — embed your query, find nearest image embeddings

Classify images zero-shot — embed class names as text, see which is closest

Build multi-modal models — use CLIP's image encoder as the "eyes" for an LLM

python

Zero-shot classification with CLIP

import clip

image = load_image("mystery_photo.jpg")
labels = ["a cat", "a dog", "a car", "a building"]
image_features = clip.encode_image(image)
text_features = clip.encode_text(labels)

Find closest match

similarities = image_features @ text_features.T
best_match = labels[similarities.argmax()]

→ "a cat"

📊

How GPT-4V Sees: The Full Pipeline

GPT-4V (and similar models like Claude's vision, Gemini) combine a vision encoder with the language model. Here's the full pipeline:

Architecture

Image → Vision Encoder (CLIP/ViT) → Projection Layer → LLM
Text  → Tokenizer → Embedding Layer ────────────────→ LLM
↓
                                                   Output Tokens

Step by step:

Image enters the vision encoder — Usually a ViT (Vision Transformer) pre-trained with CLIP. This converts the image into a sequence of rich feature vectors.

Projection layer — The vision encoder's output dimension might not match the LLM's embedding dimension. A projection layer (linear or MLP) maps vision features into the LLM's space.

Interleaving — The projected image tokens get inserted into the sequence alongside text tokens:

Sequence: [, , ..., ,

"What", "is", "shown", "in", "this", "image", "?"]

The LLM processes everything together — Self-attention operates over both image and text tokens. The model can attend from text tokens to image tokens and vice versa. This is how it "understands" the image in context.

Output is text — The model generates text tokens autoregressively, same as always. But now its predictions are informed by both the text prompt and the image content.

Token Counts

This is why vision queries are expensive:

GPT-4V's "high detail" mode actually processes the image at multiple resolutions — a low-res overview plus high-res crops of different regions — to capture both global context and fine details. This can use thousands of tokens for a single image.

Vision Transformers (ViT): The Eyes

The Vision Transformer (ViT) is the standard architecture for image encoding in multi-modal systems.

ViT vs CNN

Traditional computer vision used CNNs (Convolutional Neural Networks) — specialized architectures with convolutional filters that slide across images. ViT proved that a standard transformer, with minimal modifications, can match or beat CNNs:

Feature CNN (ResNet) ViT

The key advantage of ViT for multi-modal: since it's a transformer, its outputs are naturally compatible with LLM inputs. No awkward conversion needed.

SigLIP — The Evolution

SigLIP (Sigmoid Loss for Language-Image Pre-training) improves on CLIP by using sigmoid loss instead of softmax contrastive loss. This removes the need for large batch sizes during training and produces better image representations. Many modern multi-modal models (Gemini, PaLI) use SigLIP-based vision encoders.

Multi-modal Generation: Creating Images

So far we've talked about understanding images (vision → text). What about generating them (text → vision)?

This is where models like DALL-E, Stable Diffusion, and Midjourney come in. They take a different approach:

Approach 1: Diffusion (Separate Model)

Most current systems use a separate diffusion model for image generation (covered in Episode 60). The LLM generates a text description, which is fed to the diffusion model.

Approach 2: Native Multi-modal Generation

Some newer models generate images natively within the same architecture using discrete visual tokens:

Text tokens → LLM → Visual tokens → Image decoder → Image

"A sunset over [V_001][V_445] Image decoder Rendered

Mumbai skyline" [V_887][V_023] (trained to map sunset

[V_156]...       visual tokens     photo
                                     to pixels)

Gemini and Chameleon (Meta) take this approach — they tokenize images into discrete codes using a VQ-VAE (Vector Quantized Variational Autoencoder), then the LLM predicts these visual tokens just like it predicts text tokens.

🔬

Audio: The Third Modality

Audio follows a similar pattern to images:

Speech → Tokens

Audio waveform → Convert to spectrogram (frequency vs time image)

Spectrogram → Feed through audio encoder (like Whisper's encoder)

Audio embeddings → Project into LLM's embedding space

Interleave with text → LLM processes everything together

Audio: [wave_token_1, wave_token_2, ..., wave_token_N]

Text: "Transcribe this audio:"

Combined: [text_tokens..., audio_tokens...]

Output: "The speaker said: Hello, how are you today?"

Whisper (OpenAI) is the most widely used speech encoder. It converts audio into text with remarkable accuracy across 99 languages — the "CLIP of audio."

Real-time Voice Models

GPT-4o's voice mode takes this further — it processes audio tokens directly and generates audio tokens as output, without going through text as an intermediate step. This enables natural conversation with emotion, tone, and even singing.

🛡️

The Universal Sequence Model

The endgame of multi-modal AI is a universal sequence model — one architecture that handles all modalities natively:

Input: [text_tokens] + [image_tokens] + [audio_tokens] + [video_tokens]

↓                 ↓                ↓                ↓
      Tokenizer         ViT Encoder     Audio Encoder    Frame Sampler
          ↓                 ↓                ↓                ↓
      Embeddings        Embeddings       Embeddings       Embeddings
          ↓                 ↓                ↓                ↓
      ──────────── Single Transformer ────────────────
                            ↓
      Output: [text_tokens] or [image_tokens] or [audio_tokens]

Gemini 2.0 is closest to this vision today. It can take text, images, audio, and video as input, and generate text, images, and audio as output — all within one model.

📦

Practical Takeaways

Images become tokens — patches are extracted, embedded, and fed into the same transformer as text

CLIP bridges vision and language — it creates a shared embedding space for both modalities

Image inputs are token-expensive — a single high-res image can use 4,000-8,000 tokens of context

Vision Transformers replaced CNNs — standard transformer architecture works for images with minimal changes

Audio follows the same pattern — spectrograms get encoded into embeddings and interleaved with text

The future is universal models — single architectures handling all modalities natively

🚀

What's Next?

Episode 59: Voice & TTS — How does speech synthesis actually work? What makes Whisper so good at transcription? How do ElevenLabs and CosyVoice clone voices? We'll break down the full pipeline from text to human-sounding speech.

← Previous

Ep 57: Mixture of Experts

Next →

Ep 59: Voice & TTS

Next: Episode 59 — Voice & TTS

Close your eyes and listen to someone speak. Not the words — the way they speak. AI is learning to replicate all of that.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.