llama.cpp: The Project That Made Local AI Possible

In March 2023, a Bulgarian developer named Georgi Gerganov did something that changed AI forever. He took Meta's LLaMA model — designed to run on clusters of high-end GPUs — and made it run on a MacBook.

No Python. No PyTorch. No CUDA SDK. No cloud. Pure C/C++, compiled and running locally.

Within weeks, thousands of developers were running AI models on their laptops. The open-source AI revolution had its engine.

That engine is llama.cpp.

Why llama.cpp Matters

Before llama.cpp, running an LLM locally meant:

1. Install Python 3.10+
2. Install PyTorch (2+ GB)
3. Install CUDA Toolkit (4+ GB)
4. Install transformers, accelerate, bitsandbytes...
5. Download model weights
6. Write Python script
7. Pray that version conflicts don't kill you

llama.cpp:

1. Download the compiled binary (or compile from source)
2. Download a GGUF model file
3. Run it

That's it. One binary. One model file. No Python. No dependency hell. It compiles to native code on Mac (Metal), Linux (CUDA/CPU), and Windows (CUDA/CPU).

How It Works: The Key Innovations

1. GGUF: One File, Everything Included

From Episode 9, you know GGUF is the file format. But it's more than just quantized weights. A GGUF file contains:

GGUF file contents:

├── Magic number (GGUF header)

├── Metadata

│ ├── Architecture (llama, mistral, phi, etc.)

│ ├── Context length

│ ├── Embedding size

│ ├── Number of layers

│ ├── Head count

│ └── Quantization type

├── Tokenizer

│ ├── Vocabulary

│ ├── Merge rules (BPE)

│ └── Special tokens

└── Tensor data

├── Layer 0 weights (quantized)
    ├── Layer 1 weights (quantized)
    └── ... all layers

Everything in one file. No separate tokenizer configs, no scattered JSON files, no "download 8 files and put them in the right directory." One .gguf file and you're done.

1. mmap: Loading Without Loading

This is the magic trick that makes llama.cpp feel instant.

Normally, loading a 35 GB model means reading 35 GB from disk into RAM. That takes 30-60 seconds. llama.cpp uses mmap (memory-mapped files) instead.

Traditional loading:

Disk → [Read 35 GB into RAM] → Ready

Time: 30-60 seconds, needs 35 GB free RAM

mmap loading:

Disk → [Map file into virtual address space] → Ready

Time: < 1 second, pages loaded on demand

With mmap, the operating system treats the file on disk AS IF it were already in RAM. When the model needs a specific layer's weights, the OS loads just that page from disk. Pages that haven't been accessed yet? They stay on disk until needed.

Benefits:

Near-instant startup — no waiting for the full model to load

Shared across processes — two instances of the same model share the same physical pages

Exceeds physical RAM — a 35 GB model can "run" on 16 GB RAM (though with performance penalties from page faults)

1. Layer-by-Layer GPU Offloading

Not everyone has enough VRAM for the entire model. llama.cpp lets you offload specific layers to GPU while keeping the rest on CPU:

bash

All on CPU

./llama-cli -m model.gguf -ngl 0

All on GPU (if it fits)

./llama-cli -m model.gguf -ngl 99

20 layers on GPU, rest on CPU

./llama-cli -m model.gguf -ngl 20

The -ngl (number of GPU layers) flag is the key. Each layer you move to GPU speeds up that layer's computation. The optimal number depends on your VRAM:

70B Q4_K_M model: ~35 GB, 80 layers

RTX 4090 (24GB VRAM):

~55 layers fit in VRAM → -ngl 55

Remaining 25 layers run on CPU

Speed: ~5-8 tok/s (hybrid)

RTX 3060 (12GB VRAM):

~27 layers fit → -ngl 27

Speed: ~2-4 tok/s (mostly CPU-bound)

Full GPU (80GB A100):

All 80 layers → -ngl 80

Speed: ~15-20 tok/s

1. Metal Support (Apple Silicon)

One of llama.cpp's biggest contributions: first-class Apple Metal support. On M1/M2/M3/M4 Macs, it uses the GPU cores through Metal API — no CUDA needed.

bash

Compile with Metal support (default on macOS)

cmake -B build

cmake --build build

Metal is auto-detected. GPU layers just work:

./build/bin/llama-cli -m model.gguf -ngl 99

This made every Mac a viable AI development machine. A MacBook Air M2 with 24 GB runs 7B models at 20+ tokens/second.

🔧

Compilation: Building From Source

bash

Clone the repo

git clone https://github.com/ggerganov/llama.cpp

cd llama.cpp

macOS (Metal, automatic)

cmake -B build

cmake --build build --config Release

Linux with CUDA

cmake -B build -DGGML_CUDA=ON

cmake --build build --config Release

Linux CPU only

cmake -B build

cmake --build build --config Release

Build output (the binaries you'll use):

build/bin/

├── llama-cli # Interactive chat

├── llama-server # OpenAI-compatible API server

├── llama-bench # Benchmarking

├── llama-perplexity # Quality measurement

├── llama-quantize # Convert/quantize models

└── llama-gguf # GGUF file inspection

📊

Basic Usage

Interactive Chat

bash

./build/bin/llama-cli \

-m ~/models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \

-ngl 99 \

-c 4096 \

--chat-template llama3 \

-p "You are a helpful assistant."

Key flags:

Flag What It Does Example

API Server (OpenAI-Compatible)

This is the game-changer for developers. llama.cpp includes a server that speaks the OpenAI API:

bash

./build/bin/llama-server \

-m ~/models/llama-3.1-8b-instruct-q4_k_m.gguf \

-ngl 99 \

-c 4096 \

--host 0.0.0.0 \

--port 8080

Now you can use it with any OpenAI-compatible client:

python

from openai import OpenAI

Point to local llama.cpp server

client = OpenAI(
base_url="http://localhost:8080/v1",
    api_key="not-needed"

)

response = client.chat.completions.create(
model="local-model",
    messages=[{"role": "user", "content": "Hello!"}]

)

print(response.choices[0].message.content)

Your existing OpenAI code works with zero changes. Just swap the base URL.

Performance: What to Expect

Real-world benchmarks on common hardware:

Hardware Model Quantization Prompt Processing Generation

Notice two different numbers:

Prompt processing (prefill): How fast the model reads your input. Parallel, so GPU shines.

Generation (decode): How fast it produces tokens. Sequential, so memory bandwidth is the bottleneck (see Episode 8).

The Ecosystem It Built

llama.cpp didn't just enable local inference. It spawned an entire ecosystem:

Project What It Does Relationship to llama.cpp

When someone says "I run models locally with Ollama" — under the hood, they're running llama.cpp.

🔬

Try It Yourself — Five-Minute Setup

The absolute fastest way to get started:

bash

Option 1: Ollama (wraps llama.cpp, easiest)

curl -fsSL https://ollama.ai/install.shsh

ollama run llama3.1:8b

Option 2: llama.cpp directly (more control)

macOS

brew install llama.cpp

Download a model

wget https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf

Run it

llama-cli -m Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 99 -c 4096 --chat-template llama3

You should see output within seconds. That's a full language model running entirely on your machine. No cloud. No API key. No subscription.

🛡️

Practical Takeaways

llama.cpp made local AI accessible — one binary, one model file, no dependency hell

GGUF is the standard format — self-contained file with weights, tokenizer, and metadata

mmap makes startup instant — the model doesn't need to "load" into RAM upfront

-ngl controls GPU offloading — put as many layers on GPU as your VRAM allows

The built-in server is OpenAI-compatible — swap base_url and your existing code works

Everything from Ollama to LM Studio runs on llama.cpp underneath — learn it once, understand the ecosystem

📦

What's Next?

Episode 11: Ollama vs vLLM vs TGI — llama.cpp is the engine. But what's the best car built on top of it? Ollama for local dev, vLLM for production, TGI for Hugging Face deployments. When to use each, what they trade off, and real benchmarks.

← Previous

Ep 9: Quantization

Next →

Ep 11: Ollama vs vLLM vs TGI

Next: Episode 11 — Ollama vs vLLM vs TGI

Your boss says: put this in production, serve 100 users simultaneously. llama.cpp alone won't cut it. You need an inference engine.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.