Mixture of Experts: How GPT-4 and Mixtral Cheat at Being Smart

What if I told you GPT-4 doesn't use all its parameters when answering your question?

That 1.8 trillion parameter model? When you ask it "what's the capital of France" — only about 200 billion parameters actually fire. The rest sit there, doing nothing. Sleeping on the job.

And that's not a bug. That's the entire point.

This is Mixture of Experts (MoE) — the architecture trick that lets you build massive models without the massive compute bill. Mixtral 8x7B has 47 billion parameters but runs like a 13B model. GPT-4 is rumored to be 8 experts stitched together. DeepSeek-V3 uses 671B params but activates only 37B per token.

Welcome to the most important efficiency trick in modern AI.

The Dense Model Problem

Traditional models are dense — every token flows through every parameter. A 70B model uses all 70 billion parameters for every single prediction.

Think of it like a hospital where every doctor examines every patient. The dermatologist checks your broken leg. The cardiologist weighs in on your acne. Every doctor touches every case, whether they're relevant or not.

That's wasteful. And expensive.

Model Type Total Params Active Params per Token Compute Cost

MoE gives you the knowledge of a huge model with the compute of a small one. That's the magic.

How MoE Works

The architecture is surprisingly simple. Take a normal transformer, but replace each Feed-Forward Network (FFN) layer with multiple copies — called experts — and add a router that picks which experts to use.

The Three Components

1. Experts — Each expert is just a regular FFN layer. Same architecture, different weights. Think of them as specialists who learned different things during training.

Expert 0: FFN (hidden_dim → intermediate_dim → hidden_dim)

Expert 1: FFN (hidden_dim → intermediate_dim → hidden_dim)

Expert 2: FFN (hidden_dim → intermediate_dim → hidden_dim)

...

Expert 7: FFN (hidden_dim → intermediate_dim → hidden_dim)

1. Router (Gating Network) — A small neural network that looks at each token and decides: "which experts should handle this?" It outputs a probability distribution over all experts.

python

Simplified router logic

def route(token_embedding):

# Linear layer: maps token to expert scores
    scores = linear(token_embedding)  # shape: [num_experts]
    # Pick top-k experts
    top_k_indices = topk(scores, k=2)
    # Softmax over selected experts for weights
    weights = softmax(scores[top_k_indices])
    return top_k_indices, weights

1. Sparse Activation — Only the top-K experts process the token. K is usually 2 out of 8 experts. The outputs are weighted and combined.

Token: "photosynthesis"

Router decides: Expert 3 (weight 0.65) + Expert 7 (weight 0.35)

Output = 0.65  Expert3("photosynthesis") + 0.35  Expert7("photosynthesis")

The other 6 experts? They don't even see this token. Zero compute. Zero memory bandwidth for their activations.

🔧

The Router — The Brain Behind the Brains

The router is where the real magic happens. It's a tiny network — just a single linear layer — that makes the most consequential decision in the forward pass: who handles what?

What Do Experts Actually Specialize In?

Here's where it gets fascinating. Nobody tells the experts what to specialize in. The specialization emerges during training.

Research on Mixtral's expert routing patterns found:

Expert Tends to Activate For

But it's not clean-cut. Experts overlap. Some tokens activate the same pair of experts every time. Others are more variable. The routing is token-level, not sentence-level — different words in the same sentence might go to completely different expert pairs.

Load Balancing — The Hard Problem

What if the router sends all tokens to Expert 3 and ignores Expert 6? That's the load balancing problem. An underused expert wastes parameters. An overloaded expert becomes a bottleneck.

The fix: auxiliary load balancing loss. During training, you add a penalty term that punishes the model when experts have unequal loads:

python

Load balancing loss (simplified)

f_i = fraction of tokens routed to expert i

P_i = average router probability for expert i

balance_loss = num_experts  sum(f_i  P_i for i in range(num_experts))

This is minimized when all experts get equal traffic

This loss is added to the normal language modeling loss with a small coefficient (usually 0.01). It gently pushes the router toward even distribution without destroying the natural specialization.

📊

Mixtral 8x7B — The MoE That Changed Everything

Mistral AI's Mixtral 8x7B (December 2023) proved that MoE could be practical, open-source, and run on consumer hardware.

Architecture Breakdown

Model: Mixtral 8x7B-Instruct

Total parameters: 46.7B

Active parameters: 12.9B (per token)

Experts per layer: 8

Active experts per token: 2

Layers: 32

Hidden dim: 4096

Attention heads: 32

Context window: 32K tokens

The attention layers are shared across all experts — they're not duplicated. Only the FFN layers are split into experts. Since FFN layers make up about 2/3 of a transformer's parameters, this means:

Shared attention params: ~13B

Expert FFN params: 8 × ~4.2B = ~33.7B

Total: ~46.7B

Active per token: 13B (attention) + 2 × 4.2B (two FFN experts) ≈ 12.9B

Performance vs Cost

Mixtral matches or beats a dense 70B model while running at roughly the speed and memory cost of a 13B model. That's a 5x efficiency gain.

GPT-4 and the MoE Rumors

GPT-4's architecture was never officially confirmed, but multiple leaks and analyses suggest it's an MoE model with approximately:

8 experts per MoE layer

~220B parameters per expert

~1.8 trillion total parameters

2 experts active per token (~440B active)

16 attention heads with a massive context window

If true, this explains why GPT-4 was both dramatically better than GPT-3.5 AND feasible to serve at scale. A dense 1.8T model would be economically impossible to run for millions of users. MoE makes it viable.

DeepSeek-V3: MoE at Extreme Scale

DeepSeek-V3 pushed MoE further with 671B total params but only 37B active:

Total parameters: 671B

Active parameters: 37B

Experts per layer: 256 (!)

Active experts per token: 8

Plus 1 shared expert always active

Yes, 256 experts per layer. This is called fine-grained experts — instead of 8 big experts, you have 256 small ones. The advantage: more precise routing. The router can cherry-pick exactly the right combination of narrow specialists.

DeepSeek also introduced auxiliary-loss-free load balancing — they found a way to balance expert loads without the auxiliary loss term, using a bias mechanism in the router instead. This leads to better overall model quality.

🔬

MoE Challenges

MoE isn't free. There are real engineering tradeoffs:

1. Memory vs Compute

All experts must be in memory, even though only K are active. Your VRAM holds 47B parameters for Mixtral, but only 13B do useful work per token.

VRAM: proportional to TOTAL params (46.7B)

Compute: proportional to ACTIVE params (12.9B)

This means MoE models need more memory than their effective compute suggests.

1. Communication Overhead (Multi-GPU)

When experts live on different GPUs, tokens need to be routed across the network — the famous All-to-All communication pattern. This is the primary bottleneck for MoE at scale.

GPU 0: Experts 0, 1

GPU 1: Experts 2, 3

GPU 2: Experts 4, 5

GPU 3: Experts 6, 7

Token needs Expert 3 + Expert 5 → must talk to GPU 1 AND GPU 2

1. Training Instability

Expert routers can collapse — sending everything to one expert. Or experts can oscillate — changing specializations during training. This requires careful hyperparameter tuning and the load balancing mechanisms we discussed.

1. Batch Efficiency

Different tokens in a batch might route to different experts. This creates uneven workloads across experts within a single forward pass, reducing GPU utilization.

🛡️

The Future: MoE Everywhere

The trend is clear:

Year Model Approach

Dense models are becoming the exception. When you can have the knowledge of a trillion-parameter model at the cost of running a 50B model, there's no reason not to go MoE.

📦

Practical Takeaways

MoE = big model knowledge, small model compute — only a fraction of parameters activate per token

Routers decide which experts handle each token — specialization emerges naturally during training

Mixtral 8x7B proved MoE works in practice — matching 70B dense at 13B compute cost

VRAM scales with total params, compute with active params — you need memory for all experts but only compute a few

Load balancing is the key challenge — uneven expert usage wastes capacity

The industry is moving to MoE — GPT-4, DeepSeek-V3, and future models all use it

🚀

What's Next?

Episode 58: Multi-modal AI — How do vision models process images? What are CLIP, vision encoders, and image tokens? How does GPT-4V actually "see" your screenshots? We'll break down how AI went from text-only to understanding images, video, and audio.

← Previous

Ep 56: Monitoring & Observability

Next →

Ep 58: Multi-modal AI

Next: Episode 58 — Multi-modal AI

When you upload a screenshot to ChatGPT and it reads the text and identifies UI elements — what's actually happening under the hood?

This is part of a 98-episode series covering AI engineering from tokens to production deployment.