LoRA: Fine-Tune 0.1% of the Model, Get 95% of the Results

You want to fine-tune Llama 3 70B. Cool. Let's check the requirements:

Parameters to train: 70 billion

GPU memory needed: ~280 GB (just for the model in fp32)

Optimizer states: Another ~560 GB

Total: ~840 GB of VRAM

Hardware: 10× A100 80GB GPUs

Cost: ₹15-20 lakhs per training run

Or... you could use LoRA and do it on a single GPU with 24 GB of VRAM.

Same model. Same quality. 0.1% of the parameters. This isn't a trick — it's linear algebra.

The Core Problem: Full Fine-Tuning Is Wasteful

When you fine-tune a model, you're updating every single parameter. All 70 billion of them. But here's the thing — most of those updates are redundant.

Research from Microsoft showed something wild: the weight updates during fine-tuning have very low intrinsic rank. In plain English: the changes you make to a 70B model to teach it your task could be compressed into a much, much smaller set of numbers.

Think of it this way. You have a 10,000 × 10,000 matrix (100 million numbers). After fine-tuning, the change to this matrix — the delta — looks like it was generated by multiplying a 10,000 × 8 matrix with an 8 × 10,000 matrix.

100 million numbers of change, captured by 160,000 numbers. That's the low-rank insight.

What LoRA Actually Does

LoRA stands for Low-Rank Adaptation. Instead of modifying the original weight matrix W, you freeze it and add a small "adapter" alongside it.

The Math (Keep It Simple)

Original forward pass:

output = input × W

LoRA forward pass:

output = input × W + input × A × B

Where:

W = original frozen weights (e.g., 4096 × 4096 = 16.7M parameters)
A = small matrix (4096 × r) — initialized randomly
B = small matrix (r × 4096) — initialized to zeros
r = rank (typically 8, 16, 32, or 64)

With r = 16:

A: 4096 × 16 = 65,536 parameters

B: 16 × 4096 = 65,536 parameters

Total LoRA params: 131,072

Original W: 16,777,216 parameters

Compression: 0.78% of the original

You're training 131K numbers instead of 16.7M. And this is just ONE layer. Across the whole model, LoRA typically trains 0.1-1% of total parameters.

Why Initialize B to Zeros?

When training starts, the LoRA output is:

input × A × B = input × A × 0 = 0

So the model starts EXACTLY as the pretrained model. No change at all. Then gradually, B learns non-zero values, and the adapter starts contributing. This means LoRA can never make the model worse than the original at the start — it only adds learned behavior.

🔧

The Rank Parameter: The Only Knob That Matters

The rank r is LoRA's most important hyperparameter. It controls how much capacity the adapter has:

Rule of thumb: Start with r=16. If results are bad, go to 32. Going above 64 almost never helps.

Why Not Just Use a Huge Rank?

Because the whole point of "low rank" is that fine-tuning changes ARE low rank. If you use r=4096, you're back to full fine-tuning with extra steps. The research shows that r=8 to r=64 captures 95-99% of the information in the full weight update.

📊

What Are Adapters?

The LoRA matrices A and B together form an adapter. This is a small file (typically 10-100 MB) that sits alongside the base model.

Base Model (Llama 3 70B): 140 GB

LoRA Adapter (your fine-tune): 50 MB

Total: 140.05 GB

Adapters are incredibly powerful because:

1. Swappable

You can have multiple adapters for different tasks:

base_model + customer_support_adapter → Customer support bot

base_model + code_review_adapter → Code reviewer

base_model + medical_adapter → Medical assistant

Same base model, different behaviors. Load and unload adapters in seconds.

1. Shareable

Upload a 50 MB adapter to HuggingFace instead of a 140 GB model. Anyone with the same base model can use your fine-tune.

1. Mergeable

You can merge the adapter back into the base model:

python

Merge LoRA into base model

merged_model = model.merge_and_unload()

Now it's a regular model with LoRA baked in

merged_model.save_pretrained("merged-model")

After merging, there's ZERO inference overhead. The model runs at exactly the same speed as the original.

LoRA in Practice

Training with PEFT (HuggingFace)

python

from peft import LoraConfig, get_peft_model

from transformers import AutoModelForCausalLM

Load base model

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B")

Configure LoRA

lora_config = LoraConfig(
r=16,                          # Rank
    lora_alpha=32,                 # Scaling factor (usually 2×r)
    target_modules=[               # Which layers to adapt
        "q_proj", "k_proj",        # Attention queries and keys
        "v_proj", "o_proj",        # Attention values and output
        "gate_proj", "up_proj",    # MLP layers
        "down_proj"
    ],
    lora_dropout=0.05,             # Regularization
    bias="none",                   # Don't train biases
    task_type="CAUSAL_LM"

)

Apply LoRA

model = get_peft_model(model, lora_config)

Check trainable parameters

model.print_trainable_parameters()

# Output: trainable params: 13,631,488all params: 8,043,631,6160.17%

0.17% of parameters. That's it.

What's lora_alpha?

The scaling factor controls how much the adapter influences the output:

output = input × W + (alpha/r) × input × A × B

With alpha=32 and r=16, the scaling is 32/16 = 2. This means the adapter's contribution is amplified 2×. Higher alpha = stronger adapter influence. The conventional choice is alpha = 2 × r.

Which Layers to Target?

Not all layers benefit equally from LoRA:

For simple tasks (classification, formatting), just target q_proj and v_proj. For complex tasks (domain expertise, reasoning), target all attention + MLP layers.

Memory Savings

Here's why LoRA matters for the GPU-poor:

Approach Model Params Trainable Params VRAM Needed Hardware

The memory savings come from two things:

Fewer trainable parameters → smaller optimizer states (Adam stores 2 copies of each trainable param)

Frozen base model → can be loaded in lower precision (4-bit, 8-bit) since gradients aren't needed

🔬

When LoRA Falls Short

LoRA isn't magic. It has limitations:

Large domain shifts — If you're trying to teach an English model Japanese, the changes aren't low-rank. You might need a higher rank (64+) or full fine-tuning.

Very small models — A 1B model doesn't have much redundancy. LoRA on small models can underperform.

Knowledge injection — LoRA is great for behavior but limited for injecting new factual knowledge. That's still RAG's territory.

Inference overhead (without merging) — Running LoRA without merging adds ~5-10% latency due to the extra matrix multiplications. Always merge for production.

🛡️

Practical Takeaways

LoRA trains 0.1-1% of parameters and achieves 95%+ of full fine-tuning quality

Rank 16 is the sweet spot for most tasks — start there

Adapters are tiny files (10-100 MB) that can be swapped, shared, and merged

Always merge adapters for production — eliminates inference overhead

Target attention layers (q, k, v, o) at minimum — add MLP layers for complex tasks

LoRA makes fine-tuning accessible — an RTX 4090 can fine-tune a 70B model with QLoRA

📦

What's Next?

Episode 45: QLoRA — LoRA brought fine-tuning to single GPUs. QLoRA brings it to CONSUMER GPUs. By quantizing the base model to 4-bit and running LoRA on top, you can fine-tune a 70B model on a single RTX 3090. The math behind it is beautiful.

← Previous

Ep 43: When to Fine-Tune vs When to RAG

Next →

Ep 45: QLoRA

Next: Episode 45 — QLoRA

LoRA needs an A100. You've got an RTX 3090 in your gaming PC. QLoRA bridges the gap — fine-tune 70B on consumer hardware.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.