Edge Deployment: Running AI on Your Phone, Raspberry Pi, and Browser

What if AI didn't need the cloud?

Right now, when you ask ChatGPT a question, your words travel from your phone → through the internet → to a data center in Iowa → hit an NVIDIA H100 GPU → the answer travels back. Round trip: 500ms to 3 seconds. And you need WiFi.

But what if the model ran on your phone? No internet. No latency. No sending your private messages to a server. Your data never leaves your device.

That's edge deployment — running AI models on consumer hardware instead of cloud GPUs. And it's not a future concept. It's happening right now, on devices you already own.

Why Edge Matters

The cloud is powerful. But it has problems that edge deployment solves:

Problem Cloud Edge

For some use cases, edge isn't optional — it's the only option:

Medical devices that can't send patient data to external servers

Military/defense systems that operate without connectivity

Industrial IoT in factories with no reliable internet

Privacy-sensitive apps where users won't accept cloud processing

Developing regions with unreliable or expensive internet

Running Models on Phones

Modern smartphones are surprisingly capable AI devices.

Apple Silicon (iPhone/iPad)

Apple's Neural Engine is specifically designed for ML inference:

Device Neural Engine RAM What It Can Run

Apple's Core ML framework lets you convert and run models natively:

bash

Convert a Hugging Face model to Core ML format

pip install coremltools

python -c "

import coremltools as ct

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('microsoft/phi-3-mini-4k-instruct')

Convert to Core ML with quantization

mlmodel = ct.convert(model, compute_precision=ct.precision.FLOAT16)

mlmodel.save('phi3-mini.mlpackage')

"

Apple Intelligence runs a ~3B parameter model entirely on-device for writing tools, notification summaries, and Siri's understanding layer. No cloud required for those features.

Android (Qualcomm/MediaTek)

Android's AI story is more fragmented but rapidly improving:

Google's MediaPipe LLM Inference API provides a unified way to run models on Android:

kotlin

// Android — MediaPipe LLM

val llmInference = LlmInference.createFromOptions(

context,
    LlmInference.LlmInferenceOptions.builder()
        .setModelPath("/path/to/gemma-2b-it-gpu-int4.bin")
        .setMaxTokens(1024)
        .build()

)

val response = llmInference.generateResponse("Explain quantum computing simply")

Real-World Performance on Phones

What does "running a model on a phone" actually feel like?

Model Device Tokens/sec Feel

For context: reading speed is ~4 tokens/second. Anything above 10 tok/s feels real-time.

🔧

Raspberry Pi: The Cheapest AI Server

A Raspberry Pi 5 costs ₹5,000 ($60). It has 8GB RAM and runs Linux. That's enough to run small language models.

Setup with llama.cpp

bash

On Raspberry Pi 5 (8GB)

sudo apt update && sudo apt install -y cmake build-essential

Clone and build llama.cpp

git clone https://github.com/ggerganov/llama.cpp

cd llama.cpp

cmake -B build

cmake --build build --config Release -j4

Download a small quantized model

wget https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf

Run it

./build/bin/llama-cli -m tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf \

-p "You are a helpful assistant.\n\nUser: What is photosynthesis?\nAssistant:" \
    -n 256 -t 4

What Actually Runs on a Pi

Model Quantization RAM Usage Tokens/sec Verdict

Best use case for Pi: Always-on local AI for home automation, IoT classification, smart home assistants. Not for interactive chat with big models.

Pi Use Case: Local Smart Home AI

python

Home automation assistant on Raspberry Pi

from llama_cpp import Llama

llm = Llama(
model_path="./llama-3.2-1b.Q4_K_M.gguf",
    n_ctx=2048,
    n_threads=4

)

def classify_command(text):

"""Classify voice commands for home automation"""
    response = llm.create_chat_completion(
        messages=[{
            "role": "system",
            "content": "Classify the user command into: lights, thermostat, music, alarm, other. Reply with ONLY the category."
        }, {
            "role": "user",
            "content": text
        }],
        max_tokens=10,
        temperature=0.1
    )
    return response['choices'][0]['message']['content'].strip()

Fast classification — doesn't need a big model

print(classify_command("turn off the living room lights"))  # → "lights"
print(classify_command("set temperature to 22"))  # → "thermostat"

📊

WebLLM: AI in Your Browser

This is the wildest one. WebLLM runs full language models inside your web browser using WebGPU — no server, no installation, no downloads (well, one big download).

How WebGPU Makes This Possible

WebGPU is the successor to WebGL. It gives JavaScript direct access to your GPU — the same GPU that renders your web pages can now run neural networks.

Traditional: Browser → HTTP → Server → GPU → Response → Browser

WebLLM: Browser → WebGPU → Your GPU → Response (never leaves browser)

Running WebLLM

html

WebLLM Performance

Model Browser GPU Download Size Tokens/sec

The catch: First load downloads the full model (1-4 GB). After that, it's cached in the browser. WebGPU requires Chrome 113+ or Edge 113+. Safari support is still experimental.

When to Use WebLLM

Privacy-first applications (nothing leaves the browser)

Demos and prototypes (no backend needed)

Offline-capable web apps

Client-side text processing (summarization, classification)

❌ Not good for:

Users with weak GPUs or old browsers

Models larger than 8B (VRAM limits)

Mobile browsers (WebGPU support is spotty)

Production apps needing consistent performance across all users

ONNX Runtime: The Universal Edge Format

ONNX (Open Neural Network Exchange) is a model format designed for portability. Convert once, run anywhere.

PyTorch Model → ONNX Export → Run on:

├── Windows (DirectML)
                                ├── macOS (Core ML)
                                ├── Linux (CUDA / CPU)
                                ├── Android (NNAPI)
                                ├── iOS (Core ML)
                                ├── Browser (WASM)
                                └── Raspberry Pi (ARM CPU)

Converting and Running

python

Step 1: Export PyTorch model to ONNX

import torch

from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased")
dummy_input = torch.randint(0, 1000, (1, 128))

torch.onnx.export(

model,
    (dummy_input,),
    "distilbert.onnx",
    input_names=["input_ids"],
    output_names=["logits"],
    dynamic_axes={"input_ids": {0: "batch", 1: "seq_len"}}

)

Step 2: Run with ONNX Runtime (any platform)

import onnxruntime as ort

session = ort.InferenceSession("distilbert.onnx")
result = session.run(None, {"input_ids": input_ids.numpy()})

ONNX Runtime vs Other Options

Feature ONNX Runtime llama.cpp Core ML TensorRT

Platforms Everything CPU/Metal/CUDA Apple only NVIDIA only

The Edge Deployment Decision Tree

Do you need >13B parameter models?

├── Yes → Cloud (edge can't handle it well yet)

└── No

│
      Is privacy critical (data can't leave device)?
      ├── Yes → Edge (mandatory)
      └── No
          │
          Do you need offline capability?
          ├── Yes → Edge (mandatory)
          └── No
              │
              Is latency critical (<100ms)?
              ├── Yes → Edge (no network round trip)
              └── No
                  │
                  Budget-sensitive?
                  ├── Yes → Edge (no per-query cost)
                  └── No → Cloud (better models, easier)

🔬

Practical Takeaways

Edge AI is real today — phones run 1-3B models comfortably, 7-8B with quantization

Apple is ahead — Neural Engine + unified memory makes iPhones surprisingly good at inference

WebLLM is magic for demos — full LLM in a browser tab, zero server cost

Raspberry Pi works for classification and simple tasks — don't expect interactive chat with 7B+ models

ONNX Runtime is your portability layer — convert once, deploy everywhere

The decision is usually hybrid — edge for simple/private tasks, cloud for complex reasoning

🛡️

What's Next?

Episode 42: Model Gateways — When you're calling multiple LLM providers (OpenAI, Anthropic, Google, local models), how do you manage them all through one API? That's what gateways like LiteLLM and OpenRouter solve — and they're becoming essential infrastructure.

← Previous

Ep 40: Load Balancing Multiple Models

Next →

Ep 42: Model Gateways

Next: Episode 42 — Model Gateways

OpenAI for GPT-4. Claude for coding. Gemini for long context. Self-hosted Llama for cost. Managing all of them is a nightmare.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.