The AI Chip War: Everyone Wants a Piece of NVIDIA's Empire

NVIDIA is worth more than $3 trillion. More than Apple. More than any company in history.

All because they sell the picks and shovels for the AI gold rush. Every major AI model — GPT-4, Claude, Gemini, Llama — was trained on NVIDIA GPUs.

But here's the thing: nobody likes a monopoly. Not Google. Not Amazon. Not Microsoft. Not AMD. Not even the US government, who are worried about Taiwan (where NVIDIA's chips are made) being a single point of failure.

So everyone is building AI chips. Google has TPUs. Amazon has Trainium. AMD has MI300X. Intel has Gaudi. Startups like Groq and Cerebras are trying completely different architectures.

The AI chip war is the most important hardware battle since the CPU wars of the 1990s. And it's happening right now.

The Incumbent: NVIDIA

The H100

The current king of AI training:

80GB HBM3 memory

3.35 TB/s memory bandwidth

67 TFLOPS FP64989 TFLOPS FP8

$25,000-40,000 per card (if you can get them)

The H100 isn't just fast. It's the default. Every framework, every model, every optimization targets CUDA first. That ecosystem lock-in is NVIDIA's real moat.

The H200

Announced late 2023, shipping 2024:

141GB HBM3e memory (76% more than H100)

4.8 TB/s memory bandwidth

Same compute as H100, but more memory = bigger models

Blackwell (B100/B200)

Coming 2024-2025:

208 billion transistors (2 chips in one package)

192GB HBM3e memory

8 TB/s memory bandwidth

20 petaFLOPS FP8 (2.5x H100)

NVLink 5 — 1.8 TB/s chip-to-chip

NVIDIA isn't standing still. They're also building entire systems:

DGX GB200 — 36 Grace CPUs + 72 Blackwell GPUs

NVL72 — liquid-cooled rack with 72 GPUs

The Challenger: AMD MI300X

AMD has been the perennial #2 in GPUs. Their Radeon cards compete with NVIDIA's GeForce for gamers. But in AI? They've been an afterthought.

Until MI300X.

The Specs

Spec MI300X H100

Memory 192 GB HBM3 80 GB HBM3

Memory Bandwidth 5.3 TB/s 3.35 TB/s

On paper, MI300X beats H100 on memory and price. 192GB means you can fit bigger models on fewer cards.

The Software Problem

But specs aren't everything. The software stack matters more:

NVIDIA: CUDA → cuDNN → TensorRT → Triton

↓
        15 years of optimization
        Every framework works
        Every model is optimized

AMD: ROCm → MIOpen → ?

↓
      Getting better fast
      Still catching up
      "It works on NVIDIA" is the default

ROCm (AMD's CUDA competitor) has improved dramatically. PyTorch has native ROCm support. But you still hit edge cases:

Some operations don't have optimized kernels

Some models need tweaks to run

The ecosystem is smaller

When MI300X Wins

Memory-bound workloads — 192GB fits bigger models

Price-sensitive buyers — 2x the memory for half the price

Open source models — Llama, Mistral work great

Training from scratch — if you control the whole stack

When H100 Still Wins

Proprietary models — if OpenAI trained on CUDA, inference on CUDA is safest

Cutting-edge research — new papers target CUDA

Ecosystem compatibility — if it works on NVIDIA, it works

🔧

The Cloud Giants: Google, Amazon, Microsoft

Google TPU (Episode 69)

We covered this in detail, but the summary:

Best for: Large-scale training on Google Cloud

Best with: JAX, TensorFlow

Avoid if: You need flexibility, multi-cloud, or PyTorch is your religion

Google's advantage: they control the whole stack. Gemini was trained on TPUs, and it works.

Amazon Trainium & Inferentia

Amazon doesn't want to pay NVIDIA either. So they built their own:

Trainium (training):

Trn1 — 2021, decent but not competitive

Trn2 — 2024, much improved

Claim: 30-40% better price/performance than comparable GPUs

Inferentia (inference):

Inf2 — optimized for model serving

Low latency, high throughput

Integrated with SageMaker

The pitch: "Use AWS, use our chips, save money."

The reality: Works well for standard models. Custom architectures? Good luck.

Microsoft Maia

Microsoft's entry, announced late 2023:

Maia 100 — 5nm chip, 105 billion transistors

Designed for OpenAI workloads (no coincidence)

Liquid-cooled racks

Microsoft's strategy: "We power OpenAI. We know what AI workloads need. We'll build exactly that."

Status: Early. Very early. But with OpenAI as a customer, they have a path.

📊

The Intel Underdog: Gaudi

Intel watched the GPU revolution from the sidelines. They're trying to catch up with Gaudi:

Gaudi 2

96 GB HBM2e memory

Built-in 24x 100GbE networking (unique feature)

~$15,000

Gaudi 3 (2024)

128 GB HBM2e

2x Gaudi 2 performance

Claim: 50% better price/performance than H100

The Networking Advantage

Gaudi's unique feature: built-in Ethernet. Every chip has 24x 100GbE ports. No InfiniBand needed.

NVIDIA cluster: GPUs → NVLink → InfiniBand switches → expensive

Gaudi cluster: Gaudis → built-in Ethernet → simpler, cheaper

For large clusters, this matters. Networking is often the bottleneck and cost driver.

The Reality

Intel's software (PyTorch Habana) works for standard models. But:

Smaller ecosystem than CUDA or ROCm

Fewer optimizations

"Works on NVIDIA" is still the default assumption

Intel's bet: enterprises buying hardware will choose price/performance over ecosystem. It's a reasonable bet, but execution matters.

The Startups: Groq, Cerebras, SambaNova

While the giants fight with GPUs, some startups are asking: "What if we designed chips specifically for inference?"

Groq LPU

Groq (not Elon's Grok) makes the LPU — Language Processing Unit.

The insight: inference isn't training. You don't need massive parallelism. You need low latency.

How Groq Is Different

Traditional GPU/TPU:

Memory (HBM) ←→ Compute
↑

Bottleneck: Moving weights from memory to compute

Groq LPU:

Weights stay in on-chip SRAM (tens of megabytes, not gigabytes)

No HBM. No memory bottleneck.

The result: insane token generation speed.

Model Groq LPU H100 A100

10x faster than H100 for inference.

The Catch

Groq only works for models that fit in SRAM. That's:

Llama 3.1 8B ✅

Llama 3.1 70B ✅ (barely, on large systems)

GPT-4 ❌ (way too big)

Claude ❌ (way too big)

Groq is for open-source models you can run yourself. It's useless for API-only models.

Also: Groq chips are expensive, power-hungry, and only available through Groq's cloud (not for purchase).

When to Use Groq

Real-time applications (chatbots, voice agents)

Open-source models (Llama, Mistral)

When latency matters more than cost

When you can fit your model in SRAM

Cerebras

Cerebras makes the Wafer Scale Engine — a chip the size of a dinner plate.

WSE-3: 4 trillion transistors

900,000 AI cores

44 GB on-chip SRAM

21 petabytes/s memory bandwidth

Instead of connecting many small chips, Cerebras makes one giant chip. No chip-to-chip communication overhead.

Use case: Training massive models on a single device. Or inference with zero latency from memory access.

Reality: Extremely niche. Extremely expensive. But for specific workloads, unmatched.

SambaNova

SambaNova takes a different approach: reconfigurable dataflow.

Instead of fixed-function AI accelerators, their chips reconfigure themselves for each model. The hardware adapts to the software.

Claim: Better efficiency for both training and inference.

Reality: Interesting technology, but limited adoption outside specific enterprise customers.

The Battlefield Map

Vendor Chip Strengths Weaknesses Best For

🔬

Who Can Actually Challenge NVIDIA?

Short Answer: Nobody Yet

NVIDIA's moat isn't just hardware. It's:

CUDA — 15 years of software investment

Ecosystem — every tool, every framework, every model

Mindshare — "AI = NVIDIA" in most people's heads

Integration — NVIDIA doesn't just sell chips, they sell systems (DGX, networking, software)

Medium Term (2-3 years): AMD Has a Shot

AMD's MI300X is competitive on hardware. ROCm is getting better. If they can:

Match 90% of CUDA's functionality

Get PyTorch to "just work" on ROCm

Keep the price advantage

They could take 20-30% market share. Not dominance, but a real alternative.

Long Term (5+ years): The Cloud Giants

Google, Amazon, and Microsoft have the resources and the motivation. They're already building custom chips for their own workloads. Eventually, they might make them available more broadly.

But they're also NVIDIA's biggest customers. It's a delicate balance.

The Wildcard: Inference-Optimized Chips

Groq proved that specialized inference chips can be 10x faster. If the market shifts toward:

More open-source models

More on-premise inference

Latency-critical applications

Then inference-specialized chips could carve out a significant niche.

🛡️

What This Means for You

If You're Training Models

Default: NVIDIA H100/H200 — everything works

Budget constrained: AMD MI300X — 2x memory, lower price

On Google Cloud: TPU v5 — if you're in JAX/TensorFlow

If You're Running Inference

Default: NVIDIA — ecosystem, reliability

Latency critical: Groq — if your model fits

Cost sensitive: AMD or cloud TPUs

At scale: Consider custom silicon (Trainium, Inferentia) if you're locked into a cloud

If You're Buying Hardware

The landscape in 2024:

NVIDIA: Safe, expensive, available (with lead times)

AMD: Cheaper, riskier, improving fast

Intel: Cheapest, riskiest, wait for Gaudi 3 reviews

Everyone else: Cloud-only, specialized use cases

📦

The Bigger Picture

The AI chip war isn't just about speed. It's about:

Geopolitics: Taiwan makes most advanced chips. The US, China, and Europe all want domestic production.

Energy: AI data centers use more power than some countries. Efficient chips matter for the grid.

Democratization: If only NVIDIA can make good AI chips, they control AI's future. Competition matters.

Innovation: Monopolies get lazy. The chip war drives everyone to innovate faster.

🚀

Practical Takeaways

NVIDIA is still the safe default — ecosystem matters more than raw specs

AMD MI300X is a real alternative — especially for memory-bound workloads

Cloud TPUs work if you're all-in on Google — but don't expect flexibility

Groq is magic for latency — but only for models that fit in SRAM

The landscape is changing fast — what's true today may not be true in 6 months

Don't bet your company on unproven hardware — unless you have resources to handle the risk

🔄

What's Next?

Episode 71: Tokenizers Deep Dive — We've covered chips, training, and inference. Now let's go back to fundamentals. How do text tokenizers actually work? What's the difference between BPE, WordPiece, and SentencePiece? And why does "tokenization is all you need" have some truth to it?

← Previous

Ep 69: TPUs vs GPUs

Next →

Ep 71: BPE Tokenizer

Next: Episode 71 — BPE Tokenizer

You type 'unhappiness' into ChatGPT. The model doesn't see a word. It sees ['un', 'happ', 'iness']. Here's how BPE works.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.