QLoRA: Fine-Tune a 70B Model on Your Gaming GPU

LoRA made fine-tuning possible on a single A100. But an A100 costs ₹10 lakhs. You've got an RTX 3090 sitting in your gaming PC.

Can you fine-tune Llama 3 70B on 24 GB of VRAM?

With QLoRA, yes. Actually yes. Not a toy experiment — production-quality fine-tuning on consumer hardware. The paper that made this possible is one of the most impactful in recent AI history.

Let's break down how.

The Memory Problem

From Episode 44, we know LoRA only trains 0.1% of parameters. But you still need to LOAD the full base model into GPU memory:

Llama 3 70B in fp16: ~140 GB

LoRA adapter: ~50 MB

Optimizer states: ~200 MB

Total: ~140 GB

Your RTX 3090 has 24 GB. Even with LoRA, you can't fit the base model. The trainable parameters are tiny — the problem is the frozen weights.

QLoRA's insight: If the base model is frozen anyway, why keep it in full precision? Quantize it to 4-bit.

Llama 3 70B in 4-bit: ~35 GB (still too much for 24 GB)

Llama 3 8B in 4-bit: ~4.5 GB ✓

Llama 3 70B in 4-bit: Fits across 2× RTX 3090 ✓

For a 70B model on a single 24 GB card, you'd use CPU offloading. But 8B-13B models? They fit with room to spare.

The Three Innovations of QLoRA

QLoRA isn't just "quantize + LoRA." The paper introduced three specific techniques:

1. 4-bit NormalFloat (NF4)

Normal quantization (INT4) divides the number range into 16 equally spaced bins. But neural network weights aren't uniformly distributed — they follow a normal distribution (bell curve). Most weights are near zero, few are at the extremes.

NF4 creates 16 bins that are optimally spaced for normally distributed data:

INT4 bins:----------------------------

Equal spacing, wastes bins at extremes

NF4 bins:---------

More bins near zero where most weights live

This gives you better precision with the same 4 bits. The paper showed NF4 is information-theoretically optimal for normally distributed data.

1. Double Quantization

When you quantize weights, you need quantization constants — the scaling factors that map 4-bit values back to their original range. These constants are stored in fp32 (32 bits each).

For a 70B model with 64-value blocks, that's about 1 GB of quantization constants. Not huge, but not nothing.

Double quantization quantizes the quantization constants themselves:

Step 1: Quantize weights from fp16 → NF4 (with fp32 constants)

Step 2: Quantize the constants from fp32 → fp8

Memory for constants: 1 GB → 0.125 GB

Saves ~0.875 GB. On a 24 GB card, that's 3.6% more room for actual work.

1. Paged Optimizers

During training, GPU memory spikes happen. A long sequence comes in, activations eat up all the memory, and training crashes with an OOM error.

Paged optimizers use NVIDIA's unified memory to automatically page optimizer states to CPU RAM when GPU memory is full, then page them back when needed:

Normal: GPU full → OOM → crash

Paged: GPU full → page to CPU RAM → continue → page back when space frees up

It's like virtual memory for GPU training. Slower when paging happens, but it never crashes. And for fine-tuning with LoRA, the optimizer states are tiny — paging is rare and fast.

🔧

Memory Breakdown: Where Every Byte Goes

Let's trace exactly where memory goes when QLoRA fine-tunes Llama 3 8B:

Component Memory

─────────────────────────────────────────

Base model (4-bit NF4): 4.5 GB

Quantization constants (double Q): 0.06 GB

LoRA adapters (bf16): 0.03 GB

Optimizer states (Adam, bf16): 0.06 GB

Gradients: 0.03 GB

Activations (batch=1, seq=512):    ~2 GB

CUDA overhead: ~1 GB

─────────────────────────────────────────

Total: ~7.7 GB

7.7 GB. An RTX 3060 (12 GB) handles this easily. An RTX 3090 (24 GB) handles this with enough room for batch_size=4 and longer sequences.

Compare this to full fine-tuning of the same model:

Full fine-tune (fp32): ~96 GB

Full fine-tune (bf16): ~48 GB

LoRA (bf16 base):                  ~20 GB

QLoRA (4-bit base): ~7.7 GB

QLoRA is 12× more memory efficient than full fine-tuning in fp32.

📊

Does 4-bit Base Hurt Quality?

This is the obvious question. You're running the base model at 4-bit precision. LoRA adapters are computed in bf16, but they interact with 4-bit frozen weights. Shouldn't that lose quality?

The paper's results surprised everyone:

Method Model MMLU Score Memory

NF4 QLoRA matches full fine-tuning. No quality loss. Zero. The information-theoretic optimality of NF4 is doing the heavy lifting here — regular INT4 loses 0.5 points, but NF4 preserves everything.

The key insight: gradients flow through the LoRA adapters in full precision (bf16). The 4-bit base model is only used for the forward pass computation. All the actual learning happens in bf16. So the training is effectively bf16 quality, just with a compressed base.

QLoRA in Practice

Using bitsandbytes + PEFT

python

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training

4-bit quantization config

bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
    bnb_4bit_quant_type="nf4",          # NF4, not regular int4
    bnb_4bit_compute_dtype="bfloat16",   # Compute in bf16
    bnb_4bit_use_double_quant=True,      # Double quantization

)

Load model in 4-bit

model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3-8B",
    quantization_config=bnb_config,
    device_map="auto",

)

Prepare for training

model = prepare_model_for_kbit_training(model)

Add LoRA

lora_config = LoraConfig(
r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",

)

model = get_peft_model(model, lora_config)

model.print_trainable_parameters()

# trainable params: 13,631,488all params: 4,035,415,0400.34%

Notice the total params is ~4B, not ~8B — because the model is loaded in 4-bit, so the parameter count reflects the compressed size.

Using Unsloth (2× Faster)

python

from unsloth import FastLanguageModel

Load with 4-bit quantization (Unsloth handles everything)

model, tokenizer = FastLanguageModel.from_pretrained(

model_name="unsloth/llama-3-8b-bnb-4bit",
    max_seq_length=2048,
    load_in_4bit=True,

)

Add LoRA (Unsloth-optimized)

model = FastLanguageModel.get_peft_model(
model,
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                     "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0,
    bias="none",

)

Unsloth is a library that optimizes QLoRA training with custom CUDA kernels. It's genuinely 2× faster and uses 50% less memory. If you're doing QLoRA, use Unsloth.

Consumer GPU Cheat Sheet

What can you fine-tune on YOUR GPU?

GPU VRAM Max Model (QLoRA) Batch Size Notes

RTX 3060 12 GB 7-8B 1 Tight, use gradient checkpointing

RTX 3070 8 GB 3-7B 1 Limited, but works

RTX 3080 10 GB 7-8B 1 Sweet spot for 7B

RTX 3090 24 GB 13-14B 2-4 Best consumer option

RTX 4090 24 GB 13-14B 4-8 Faster than 3090

Pro tip: Use gradient_checkpointing=True to trade compute for memory. Reduces VRAM by ~30% at the cost of ~20% slower training.

🔬

The Workflow: From GPU to Production

1. Train with QLoRA (4-bit base + bf16 LoRA)

↓

1. Save the LoRA adapter (~50 MB)

↓

1. For production, choose:

a. Load base model (fp16) + merge LoRA → Full precision model

b. Load base model (GGUF Q4) + LoRA → Quantized inference

c. Upload adapter to HuggingFace → Others can use it

The beauty: you train on your RTX 3090, but deploy at whatever precision makes sense for your production hardware.

🛡️

Practical Takeaways

QLoRA = 4-bit base model + bf16 LoRA adapters — train in full precision on a quantized base

NF4 quantization loses zero quality compared to full fine-tuning — use it, not INT4

Always enable double quantization — free memory savings with no downside

RTX 3090/4090 can fine-tune 13B models comfortably — you don't need cloud GPUs

Use Unsloth — 2× faster, 50% less memory, no quality loss

Merge adapters for production — eliminates the quantization overhead for inference

📦

What's Next?

Episode 46: Training Data for Fine-Tuning — You can fine-tune on your gaming GPU now. But what data do you actually feed it? Alpaca format? ShareGPT format? Chat templates? How much data is enough? (Spoiler: quality beats quantity every single time.) The training data is 80% of your fine-tune's quality.

← Previous

Ep 44: LoRA

Next →

Ep 46: Training Data for Fine-Tuning

Next: Episode 46 — Training Data for Fine-Tuning

You fire up training and the model outputs garbage. Worse than the base model. Three days wasted. The problem? Your data.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.