Apple Silicon for AI: Why Macs Are Legit for Inference

Here's a fact that surprises most AI engineers: a Mac Studio with M4 Ultra can run Llama 3 70B unquantized in FP16 — the full 140 GB model — on a single machine. No multi-GPU setup. No NVLink. No $30,000 H100.

A machine that sits on your desk, runs silently, and costs a fraction of an NVIDIA DGX.

How? One word: unified memory. Apple Silicon doesn't have separate CPU RAM and GPU VRAM. It's all one pool. And that single architectural decision makes Macs the dark horse of AI inference.

The Unified Memory Advantage

The NVIDIA Way

Traditional GPU systems have separate memory pools:

┌─────────────┐ PCIe (64 GB/s) ┌──────────────┐

│ CPU RAM │ ←──────────────────→ │ GPU VRAM │

│ 128 GB │ │ 24-80 GB │

│ DDR5 │ │ HBM3 │

└─────────────┘ └──────────────┘

Problem: Model must fit ENTIRELY in GPU VRAM.

CPU RAM is irrelevant for inference.

70B FP16 = 140 GB → doesn't fit in any single GPU.

The Apple Way

Apple Silicon uses unified memory — one pool shared between CPU and GPU:

┌──────────────────────────────────────┐

│ Unified Memory (LPDDR5) │

│ 128 GB / 192 GB / 512 GB │

│ │

│ ┌─── CPU Cores ───┐ │

│ │ Access memory │ │

│ │ directly │ │

│ └─────────────────┘ │

│ │

│ ┌─── GPU Cores ───┐ │

│ │ Access SAME │ │

│ │ memory directly │ │

│ └─────────────────┘ │

└──────────────────────────────────────┘

70B FP16 = 140 GB → fits in 192 GB unified memory ✅

No data transfer needed. GPU reads weights directly.

No PCIe bottleneck. No data copying between CPU and GPU memory. The model weights sit in memory once and both CPU and GPU access them directly.

Apple Silicon Lineup for AI

Chip GPU Cores Memory Bandwidth Released

What Can Each Chip Run?

Chip (Max RAM) INT4 Models FP16 Models

An M4 Ultra with 512 GB can run models that would require multiple H100s on NVIDIA. The sheer memory capacity is unmatched by any single GPU.

🔧

MLX: Apple's AI Framework

MLX is Apple's machine learning framework, designed specifically for Apple Silicon. Think of it as "PyTorch for Macs."

Why Not Just Use PyTorch?

PyTorch on Mac works (via MPS — Metal Performance Shaders), but MLX is purpose-built for Apple's architecture:

Feature PyTorch MPS MLX

Unified memory Copies data True zero-copy

Running Models with MLX

bash

Install

pip install mlx mlx-lm

Download and run a model

mlx_lm.generate \

--model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit \

--prompt "Explain GPU architecture" \

--max-tokens 500

Or start a chat server

mlx_lm.server --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit

MLX Model Conversion

Most Hugging Face models can be converted to MLX format:

bash

Convert any HF model to MLX format

mlx_lm.convert \

--hf-path meta-llama/Llama-3.1-70B-Instruct \

--mlx-path ./llama-70b-mlx \

--quantize --q-bits 4

The mlx-community hub has pre-converted models

https://huggingface.co/mlx-community

📊

Metal: Apple's GPU API

While NVIDIA has CUDA, Apple has Metal — its GPU compute API:

NVIDIA stack: Apple stack:

CUDA → cuBLAS Metal → MPS → MLX

→ cuDNN → MPSGraph

→ TensorRT → Core ML

Metal Performance Shaders (MPS)

MPS provides pre-built, optimized GPU compute kernels for common operations:

Matrix multiplication

Convolution

Attention

Normalization

PyTorch's MPS backend uses these under the hood. When you do model.to("mps"), PyTorch dispatches operations to Metal Performance Shaders.

Apple Neural Engine (ANE)

Apple Silicon also includes a dedicated Neural Engine — specialized hardware for inference:

Chip ANE Performance

M4 38 TOPS

M4 Pro 38 TOPS

M4 Max 38 TOPS

M4 Ultra 76 TOPS

The ANE is designed for smaller models and is used heavily by on-device features (Siri, camera processing, autocorrect). It's less relevant for LLM inference, which primarily uses the GPU cores.

Performance: Apple vs NVIDIA

Let's be honest about where Apple Silicon wins and where NVIDIA dominates:

Inference Speed (tokens/second)

The Analysis

NVIDIA wins on speed — when the model fits. An H100 is 3-5x faster per token than Apple Silicon. HBM3 bandwidth (3,350 GB/s) crushes LPDDR5 (546-819 GB/s).

Apple wins on capacity — when the model doesn't fit on one GPU. Running 70B FP16 on a single Mac is cheaper and simpler than buying 2+ H100s.

Apple wins on cost-per-GB:

H100 80GB:    ~$30,000 → $375/GB of VRAM
Mac Studio M4 Ultra 512GB: ~$10,000 → $20/GB of unified memory

Apple is ~19x cheaper per GB of model-accessible memory.

Apple wins on power:

H100: 700W TDP

Mac Studio M4 Ultra: ~200W max

H100 electricity cost: ~$600/year (24/7)

Mac Studio: ~$175/year (24/7)

Where Apple Falls Short

Limitation Impact

Practical Apple Silicon Setups

The Developer Machine

Hardware: MacBook Pro M4 Max, 128 GB

Cost: ~$4,500

Can run:

• 70B INT4 models locally (35 GB)
• Multiple 8B models simultaneously
• Full development workflow + model serving

Use for: Development, testing, small-scale inference

The Home Server

Hardware: Mac Studio M4 Ultra, 192 GB

Cost: ~$6,000

Can run:

• 70B FP16 (140 GB) — unquantized!
• 405B INT4 (202 GB) with some context
• Multiple 70B INT4 models

Use for: Personal inference server, RAG systems, local AI

The Maximum Setup

Hardware: Mac Studio M4 Ultra, 512 GB

Cost: ~$10,000

Can run:

• 405B FP16 (810 GB — just barely with swap)
• 405B INT8 (405 GB) — comfortably
• Multiple 70B FP16 models simultaneously

Use for: Maximum local inference, no-cloud needed

🔬

Ollama on Mac: The Easiest Path

Ollama is the simplest way to run models on Apple Silicon:

bash

Install

curl -fsSL https://ollama.com/install.shsh

Run a model (downloads automatically)

ollama run llama3.1:8b

ollama run llama3.1:70b    # Needs 40+ GB RAM

ollama run mistral

ollama run codellama:34b

API server (OpenAI-compatible)

Automatically starts at http://localhost:11434

curl http://localhost:11434/v1/chat/completions \

-d '{

"model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "Hello!"}]

}'

Ollama automatically uses MLX or llama.cpp's Metal backend. Zero configuration for Apple Silicon — it just works.

🛡️

llama.cpp Metal: Maximum Performance

For the best Apple Silicon performance, llama.cpp with Metal backend gives you the most control:

bash

Build with Metal support

git clone https://github.com/ggerganov/llama.cpp

cd llama.cpp

make LLAMA_METAL=1

Run with Metal

./llama-cli -m llama-3-70b-Q4_K_M.gguf \

-ngl 99 \ # Offload all layers to GPU

-c 8192 \ # Context length

--temp 0.7 \

-p "Your prompt here"

The -ngl 99 flag is key — it tells llama.cpp to put all model layers on the GPU (Metal). Without it, layers stay on CPU and inference is much slower.

📦

Practical Takeaways

Unified memory is Apple's superpower — the entire system memory is GPU-accessible, no VRAM limits

Apple wins on capacity and cost, NVIDIA wins on speed — cheaper per GB, but slower per token

MLX is the native framework — purpose-built for Apple Silicon, zero-copy memory

70B FP16 on a single Mac — impossible on any single NVIDIA GPU, trivial on M4 Max/Ultra

Apple is best for inference, weak for training — no distributed training, limited training support

Ollama makes it dead simple — install and run, Metal acceleration automatic

🚀

What's Next?

Episode 68: Cloud GPU Pricing — AWS, GCP, Lambda Labs, RunPod, Vast.ai — who's cheapest? Spot vs on-demand pricing. How to optimize your cloud GPU bill without sacrificing performance.

← Previous

Ep 66: Multi-GPU Setups

Next →

Ep 68: Cloud GPU Pricing

Next: Episode 68 — Cloud GPU Pricing

You need GPU compute. Cloud providers all want your money. Here's the real pricing breakdown so you don't go broke.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.