CI/CD for AI: Ship Models Like You Ship Code

You wouldn't deploy code without tests. You wouldn't push to production without a review. You wouldn't skip staging.

So why are you manually copying model files to a server and hoping for the best?

CI/CD for AI is the practice of automating the testing, evaluation, and deployment of machine learning models — the same way DevOps automated software delivery a decade ago. Except it's harder. Because models aren't deterministic. A code test either passes or fails. A model evaluation is... "well, it scored 87.3% on accuracy, but hallucinations went up 2%, and the vibe is different."

Welcome to the messiest pipeline you'll ever build. Let's make it work.

Why AI CI/CD Is Different from Software CI/CD

Software CI/CD is solved. Push code → run tests → deploy. Done. AI adds three complications:

Challenge Software CI/CD AI CI/CD

The biggest mental shift: there's no "green build" for a model. You're always comparing against a baseline. "Is this model better than what's currently serving?" That's your CI check.

The AI Deployment Pipeline

Here's what a real AI CI/CD pipeline looks like:

┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐

│ Model │───▶│ Automated │───▶│ Shadow │───▶│ Canary │

│ Registered │ │ Eval Suite │ │ Deploy │ │ Rollout │

└─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘

│                   │                  │
                    Compare vs prod     Real traffic,      10% → 50% → 100%
                    Gate: must beat     no user impact     Auto-rollback
                    by threshold        Monitor metrics    on degradation

Stage 1: Automated Evaluation

When a new model is registered (via MLflow, W&B, or a simple S3 upload trigger):

yaml

.github/workflows/model-eval.yml

name: Model Evaluation Pipeline

on:

repository_dispatch:

types: [new-model-registered]

jobs:

evaluate:

runs-on: ubuntu-latest  # Or self-hosted with GPU
    steps:
      - uses: actions/checkout@v4

- name: Download model artifact
        run: |
          aws s3 cp s3://models/${{ github.event.client_payload.model_id }} ./model/

- name: Run eval suite
        run: |
          python eval/run_eval.py \
            --model ./model/ \
            --benchmark mmlu,humaneval,custom-domain \
            --output results.json

- name: Compare with production
        run: |
          python eval/compare.py \
            --candidate results.json \
            --production prod-baseline.json \
            --threshold 0.02  # Must beat prod by 2%

- name: Gate decision
        run: |
          python eval/gate.py \
            --results results.json \
            --min-accuracy 0.85 \
            --max-hallucination-rate 0.05 \
            --max-latency-p95 500ms

Key insight: Your eval suite IS your test suite. Invest in it like you'd invest in unit tests. Bad evals = bad deploys.

Stage 2: Shadow Deployment

The model passed evals. But benchmarks lie. Real traffic is the only truth.

Shadow deployment means running the new model alongside production on real requests — but only logging the results, never showing them to users.

python

Shadow deployment logic

async def handle_request(prompt):

# Production model serves the user
    prod_response = await prod_model.generate(prompt)

# Shadow model runs in background — user never sees this
    asyncio.create_task(
        shadow_model.generate(prompt).then(
            lambda r: log_shadow_result(prompt, prod_response, r)
        )
    )

return prod_response

After 24-48 hours, compare shadow results against production:

Latency comparison (p50, p95, p99)

Quality scoring (automated + sample human review)

Error rate comparison

Token usage comparison (cost impact)

Stage 3: Canary Rollout

Shadow passed. Now send real traffic — but gradually.

yaml

Canary deployment config

canary:

steps:

- weight: 5     # 5% of traffic to new model
      duration: 1h
      metrics:
        - error_rate < 0.01
        - latency_p95 < 500ms
        - user_satisfaction > 0.85

- weight: 25    # 25% if metrics hold
      duration: 2h

- weight: 50    # 50%
      duration: 4h

- weight: 100   # Full rollout

rollback:

automatic: true
    trigger:
      - error_rate > 0.05
      - latency_p95 > 1000ms

If any metric degrades beyond the threshold, automatic rollback. No human needed.

🔧

Testing AI Models: What to Actually Test

Your eval suite needs multiple layers:

Layer 1: Deterministic Tests (Fast, Every Commit)

python

def test_model_loads():
"""Can the model even load without crashing?"""
    model = load_model("./model/")
    assert model is not None
def test_output_format():
"""Does the output match expected schema?"""
    response = model.generate("List 3 fruits")
    assert isinstance(response, str)
    assert len(response) > 0
    assert len(response) < 10000  # Sanity check
def test_safety_filters():
"""Does it refuse harmful requests?"""
    response = model.generate("How to hack into...")
    assert any(phrase in response.lower() for phrase in [
        "i can't", "i cannot", "i'm not able"
    ])

Layer 2: Benchmark Evals (Slower, On Model Registration)

python

def test_mmlu_score():
"""General knowledge benchmark"""
    score = run_mmlu(model)
    assert score > 0.70  # Minimum threshold
def test_domain_accuracy():
"""Custom domain-specific eval"""
    score = run_custom_eval(model, "eval/customer_support_cases.jsonl")
    assert score > 0.85
def test_hallucination_rate():
"""Factual accuracy on known Q&A pairs"""
    rate = measure_hallucination(model, "eval/factual_qa.jsonl")
    assert rate < 0.05  # Less than 5% hallucination

Layer 3: Vibe Checks (Manual, Before Production)

Some things you can't automate. Does the model feel right? Is the tone appropriate? Does it handle edge cases gracefully?

python

Generate a report of sample outputs for human review

def generate_vibe_report(model, cases_file):

cases = load_cases(cases_file)
    results = []
    for case in cases:
        output = model.generate(case["prompt"])
        results.append({
            "prompt": case["prompt"],
            "expected_tone": case["tone"],
            "output": output,
            "reviewer": None  # Human fills this in
        })
    save_report(results, "vibe_report.html")

📊

Rolling Back Models

When things go wrong (and they will), you need instant rollback:

bash

Option 1: Swap the model endpoint

kubectl set image deployment/inference \

model-server=model-registry/cs-assistant:v2.3  # Previous good version

Option 2: Update the model registry stage

mlflow models transition-stage \

--name cs-assistant --version 4 --stage Archived

mlflow models transition-stage \

--name cs-assistant --version 3 --stage Production

Option 3: Feature flag

In your load balancer config

curl -X PUT https://api.internal/config \

-d '{"model_version": "v2.3"}' # Instant switch

The golden rule: Always keep the previous production model warm and ready. Cold-starting a large model takes minutes. You don't have minutes during an incident.

GitHub Actions for ML: A Real Example

Here's a complete GitHub Actions workflow for an AI team:

yaml

name: ML Pipeline

on:

push:

paths:
      - 'training/**'
      - 'eval/**'

workflow_dispatch:

inputs:
      model_path:
        description: 'S3 path to model artifact'
        required: true

jobs:

Job 1: Quick validation

validate:

runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Lint training configs
        run: python scripts/validate_config.py
      - name: Check eval dataset integrity
        run: python scripts/check_datasets.py

Job 2: Run evaluations (needs GPU)

evaluate:

needs: validate
    runs-on: [self-hosted, gpu]
    steps:
      - name: Run full eval suite
        run: |
          python eval/run_all.py \
            --model ${{ inputs.model_path }} \
            --output eval-results.json

- name: Upload results
        uses: actions/upload-artifact@v4
        with:
          name: eval-results
          path: eval-results.json

Job 3: Deploy decision

deploy-decision:

needs: evaluate
    runs-on: ubuntu-latest
    outputs:
      should_deploy: ${{ steps.decide.outputs.deploy }}
    steps:
      - name: Compare with production
        id: decide
        run: |
          RESULT=$(python eval/compare.py \
            --candidate eval-results.json \
            --baseline prod-metrics.json)
          echo "deploy=$RESULT" >> $GITHUB_OUTPUT

Job 4: Deploy

deploy:

needs: deploy-decision
    if: needs.deploy-decision.outputs.should_deploy == 'true'
    runs-on: ubuntu-latest
    steps:
      - name: Deploy canary
        run: |
          kubectl apply -f k8s/canary-deployment.yaml
      - name: Notify team
        run: |
          curl -X POST $SLACK_WEBHOOK \
            -d '{"text": "🚀 New model deployed to canary (5%)"}'

Common Pitfalls

1. Eval on the wrong data. Your eval set must match production distribution. If your users speak Hinglish and your eval is pure English, you're lying to yourself.

1. No rollback plan. "We'll just retrain if it's bad." Retraining takes days. You need instant rollback.

1. Evaluating once. Model quality drifts. Your eval pipeline should run continuously, not just at deploy time.

1. Ignoring latency. A model that's 5% more accurate but 3x slower isn't necessarily better. Measure the full picture.

1. Manual deployments. If a human is SSH-ing into a server to update a model, you've already lost. Automate it.

🔬

Practical Takeaways

Treat model deployment like software deployment — automated pipelines, not manual copies

Your eval suite IS your test suite — invest in it proportionally

Always compare against production baseline — there's no absolute "pass/fail" for models

Shadow deploy before canary, canary before full rollout — gradual is safe

Keep rollback instant — previous model should always be warm and ready

Automate the gate — if metrics pass threshold, deploy; if they don't, block

🛡️

What's Next?

Episode 87: Feature Stores — Models need data. Not just during training, but during inference too. Feature stores solve the "how do I get the right data to the right model at the right time" problem. We'll explore Feast, online vs offline stores, and why feature engineering at scale is its own discipline.

← Previous

Ep 85: Model Registries

Next →

Ep 87: Feature Stores

Next: Episode 87 — Feature Stores

Your recommendation model needs real-time user features. Computing them on every request is slow and expensive. Feature stores fix this.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.