Model Merging: Frankenstein Your Way to a Better Model

You have two fine-tuned models. One is amazing at coding. The other is amazing at creative writing. You want one model that does both.

The obvious approach: fine-tune on a combined dataset of coding AND creative writing examples. But that takes GPU hours, data curation, and the model might lose quality on both tasks.

The insane approach: just average their weights together.

And it works. This is model merging — the most "this shouldn't work but it does" technique in AI. The open-source community has built models that beat their individual components by literally mashing weight matrices together. No training. No GPUs. Just math.

Why Merging Works (The Intuition)

When you fine-tune two models from the same base, they start from the same point in "weight space." Each fine-tune moves the weights in a slightly different direction — one toward coding, one toward creative writing.

Coding Model
                   ↗

Base Model ──────

↘
                    Creative Model

The merged model sits somewhere between them:

Coding Model
                   ↗

Base Model ────── ──→ Merged Model

↘
                    Creative Model

Because both models started from the same base, the regions of weight space they moved to are structurally compatible. The coding knowledge lives in certain parameters, the creative knowledge lives in others. Averaging them combines both — with surprisingly little interference.

This only works when models share the same architecture AND the same base model. You can't merge Llama and Mistral (different architectures). You CAN merge two Llama fine-tunes.

Merging Methods

1. Linear Merge (SLERP)

The simplest approach: weighted average of two models.

python

Linear interpolation

merged_weight = alpha  model_A_weight + (1 - alpha)  model_B_weight
SLERP (Spherical Linear Interpolation) is a fancier version that interpolates along a sphere instead of a straight line, preserving the magnitude of weight vectors:

python

SLERP interpolation

import numpy as np

def slerp(t, v0, v1, DOT_THRESHOLD=0.9995):
"""Spherical linear interpolation between two vectors."""
    dot = np.sum(v0 * v1 / (np.linalg.norm(v0) * np.linalg.norm(v1)))
if abs(dot) > DOT_THRESHOLD:
        # Vectors are very close, use linear interpolation
        return (1 - t) * v0 + t * v1
theta_0 = np.arccos(dot)
    sin_theta_0 = np.sin(theta_0)
    theta = theta_0 * t
    sin_theta = np.sin(theta)
s0 = np.cos(theta) - dot * sin_theta / sin_theta_0
    s1 = sin_theta / sin_theta_0
return s0 * v0 + s1 * v1

Use when: Merging exactly 2 models with similar capabilities.

1. TIES (Trim, Elect Sign, Merge)

TIES addresses a key problem: when merging, many parameters fight each other. Model A wants a parameter to go up, Model B wants it to go down. Averaging gives you zero — both contributions cancelled out.

TIES fixes this in three steps:

Step 1: TRIM — Remove small-magnitude changes (they're noise)

Keep only the top-k% most significant parameter changes

Step 2: ELECT SIGN — For each parameter, vote on the direction

If 2 out of 3 models moved it positive → positive wins
         Discard the minority direction

Step 3: MERGE — Average only the parameters that agree

yaml

mergekit TIES config

models:

• model: base_model

parameters:
      weight: 1.0

• model: coding_model

parameters:
      weight: 0.5
      density: 0.5  # Keep top 50% of changes (TRIM)

• model: creative_model

parameters:
      weight: 0.5
      density: 0.5

merge_method: ties

base_model: base_model

dtype: float16

Use when: Merging 2+ models where you want to avoid interference.

1. DARE (Drop And REscale)

DARE takes a different approach: randomly drop parameter changes and rescale the remaining ones.

Step 1: For each model's delta (change from base):

- Randomly drop p% of the changes (set to zero)
        - Rescale remaining by 1/(1-p) to maintain magnitude

Step 2: Sum the surviving deltas

Step 3: Add to base model

Why does dropping work? Because of the redundancy in fine-tuning changes. You can drop 90% of parameter changes and the model barely notices — the remaining 10% captures most of the learned behavior. And by randomly dropping different parameters from each model, you reduce interference.

yaml

mergekit DARE config

models:

• model: coding_model

parameters:
      weight: 0.4
      density: 0.3  # Keep only 30% of changes

• model: creative_model

parameters:
      weight: 0.4
      density: 0.3

merge_method: dare_ties # DARE with TIES sign election

base_model: base_model

dtype: float16

Use when: Merging models with very different specializations.

1. Passthrough (Frankenmerge)

The wildest technique. Instead of merging parameter values, you stack LAYERS from different models:

Layers 1-16: From Coding Model

Layers 17-24: From Creative Model

Layers 25-32: From Coding Model

This creates a literal Frankenstein model. It shouldn't work at all — you're stitching together layers that were never trained together. But it often produces surprisingly good results, especially when both models have the same base architecture.

yaml

mergekit passthrough (frankenmerge) config

slices:

• sources:
• model: coding_model

layer_range: [0, 16]

• sources:
• model: creative_model

layer_range: [16, 24]

• sources:
• model: coding_model

layer_range: [24, 32]

merge_method: passthrough

dtype: float16

Use when: You want to experiment. Results are unpredictable but sometimes magical.

🔧

MergeKit: The Standard Tool

mergekit is the go-to library for model merging:

bash

Install

pip install mergekit

Run a merge

mergekit-yaml config.yaml ./merged-model --cuda

Full Example: Merging a Coder + a Writer

yaml

merge_config.yaml

models:

• model: ./llama3-8b-coding-lora-merged

parameters:
      weight: 0.6
      density: 0.5

• model: ./llama3-8b-creative-lora-merged

parameters:
      weight: 0.4
      density: 0.5

merge_method: dare_ties

base_model: meta-llama/Meta-Llama-3-8B

parameters:

normalize: true

int8_mask: true

dtype: float16

bash

Execute the merge (needs enough RAM to hold all models)

mergekit-yaml merge_config.yaml ./merged-coder-writer \

--cuda \
    --lazy-unpickle

Test the result

python -c "

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('./merged-coder-writer')
tokenizer = AutoTokenizer.from_pretrained('./merged-coder-writer')

Test coding

Test creative writing

Compare with individual models

"

Merging LoRA Adapters

You don't need to merge full models. You can merge LoRA adapters directly:

python

from peft import PeftModel

Load base model

base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B")

Load and merge first adapter

model = PeftModel.from_pretrained(base, "./coding-adapter")
model = model.merge_and_unload()

Load and merge second adapter (with weight)

model = PeftModel.from_pretrained(model, "./creative-adapter")

Manually scale the adapter before merging

for name, param in model.named_parameters():

if "lora" in name:
        param.data *= 0.5  # Reduce influence to 50%

model = model.merge_and_unload()

model.save_pretrained("./dual-adapter-merged")

📊

The Merge Leaderboard Phenomenon

The open-source AI community went absolutely wild with merging. On the Open LLM Leaderboard, merged models regularly outperform their components:

Model How It Was Made MMLU

The merged model is BETTER than both components AND the base model. This shouldn't happen, but it does — consistently.

When Merging Fails

Different base models — Merging a Llama fine-tune with a Mistral fine-tune produces garbage. Same architecture and same base weights required.

Too many models — Merging 5+ models usually degrades quality. Stick to 2-3.

Conflicting specializations — A model fine-tuned to be concise merged with one fine-tuned to be verbose will be confused, not balanced.

No evaluation — Merging is cheap to try but still needs evaluation. Don't ship without testing.

Wrong density/weight — These hyperparameters matter. Grid search a few values.

The Merge Workflow

1. Fine-tune Model A (coding) with LoRA
2. Fine-tune Model B (creative) with LoRA
3. Merge both adapters into base model
4. Evaluate merged model on BOTH tasks
5. If quality drops, adjust weights/density
6. Compare: merged vs individual + routing
7. Ship whichever wins

Alternative to merging: Use a router that sends coding questions to the coding model and creative questions to the creative model. Routing is more reliable but adds complexity. Merging is simpler but might lose quality on edge cases.

🔬

Practical Takeaways

Model merging combines capabilities without training — just math on weight matrices

SLERP for 2 similar models, DARE/TIES for different specializations — choose the method based on your models

Both models must share the same base — same architecture AND same pretrained weights

Merged models can outperform components — this is counterintuitive but consistently true

Always evaluate after merging — cheap to try doesn't mean guaranteed to work

MergeKit is the standard tool — YAML config, one command, done

🛡️

What's Next?

Episode 49: RLHF & DPO — Fine-tuning teaches the model your task. But how do you teach it to be helpful? How do you make it prefer good answers over technically-correct-but-terrible ones? Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are how models go from smart to useful.

← Previous

Ep 47: Fine-Tune Evaluation

Next →

Ep 49: RLHF & DPO

Next: Episode 49 — RLHF & DPO

GPT-3 was brilliant but also a nightmare. RLHF and DPO taught it to be helpful instead of just technically correct.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.