How Cursor/Copilot Works: AI Coding Assistants Under the Hood

You're typing code. You pause for half a second. Grey text appears — completing your function, writing the next 5 lines, sometimes an entire implementation. You hit Tab. It's exactly what you wanted.

How?

AI coding assistants like GitHub Copilot, Cursor, and Codeium aren't just chatbots pasted into your editor. They use fundamentally different techniques from ChatGPT — fill-in-the-middle completion, language server integration, codebase-wide context retrieval, and speculative edits that predict what you'll do next.

The result feels like magic. The engineering behind it is wild. Let's break it all down.

Fill-in-the-Middle: The Core Trick

ChatGPT generates left-to-right. Given text before the cursor, it predicts what comes after.

Coding assistants do something different: fill-in-the-middle (FIM). Given text BEFORE the cursor AND text AFTER the cursor, they predict what goes IN BETWEEN.

python

# Your code (= cursor position)

def calculate_total(items):
|
    return total

Left-to-right (ChatGPT-style) only sees:

"def calculate_total(items):\n "

Has NO IDEA the function should return 'total'

Fill-in-the-middle sees:

PREFIX: "def calculate_total(items):\n "

SUFFIX: "\n return total"

→ Must generate something that makes sense BETWEEN these

The model is trained with a special format:

def calculate_total(items):

<|fim_suffix|>
    return total<|fim_middle|>total = sum(item.price * item.quantity for item in items)

Why this matters enormously:

The suffix gives crucial context about what the code should DO

Variable names used later constrain what's generated now

Function signatures and return types guide the implementation

It reduces hallucination — the model can't invent random variable names because the suffix uses specific ones

Without FIM, the model might generate:

python

def calculate_total(items):
result = 0
    for item in items:
        result += item.cost  # Wrong variable name!
    return result  # Wrong return variable!

With FIM, it knows total is used later and generates accordingly.

Context Gathering: How It Knows Your Codebase

The biggest challenge for AI coding assistants isn't generation — it's context. Your cursor is in one file, but the relevant code might be in 20 other files.

The Context Window Problem

A model has limited context (128K tokens typically). Your codebase might be millions of tokens. What goes in the prompt?

Available context budget: ~8K-16K tokens (for fast completions)

What needs to fit:

├── Current file (around cursor) ~2K tokens

├── Imports and dependencies ~1K tokens

├── Related functions in other files ~3K tokens

├── Type definitions / interfaces ~1K tokens

├── Recent edit history ~500 tokens

├── LSP information (errors, types) ~500 tokens

└── Repository-level context ~500 tokens

How Context Is Gathered

Step 1: Nearby code — The most relevant code is the code around your cursor. Current function, current file, imports.

Step 2: Language Server Protocol (LSP) integration — This is where it gets sophisticated. Your editor already has a language server that understands your code's structure:

LSP provides:

├── Type information: "items is List[OrderItem]"

├── Function signatures: "OrderItem.price: float, OrderItem.quantity: int"

├── Error diagnostics: "TypeError on line 42: expected str, got int"

├── Symbol definitions: "total is referenced in return statement on line 15"

├── Import graph: "This file imports from utils.py, models.py"

└── Call hierarchy: "calculate_total is called by process_order in orders.py"

The coding assistant uses LSP data to find exactly which other files matter:

python

Cursor's context gathering (simplified)

def gather_context(cursor_position, file_path):

context = []

# 1. Current file (around cursor)
    context.append(get_surrounding_code(file_path, cursor_position, window=100))

# 2. LSP: Get type definitions
    symbols = lsp.get_symbols_at_cursor(cursor_position)
    for symbol in symbols:
        definition = lsp.go_to_definition(symbol)
        context.append(definition.source_code[:500])  # First 500 chars

# 3. LSP: Get callers/callees
    references = lsp.find_references(current_function)
    for ref in references[:5]:  # Top 5 most relevant
        context.append(ref.surrounding_code)

# 4. Recently edited files
    recent_files = editor.get_recently_modified(limit=5)
    for f in recent_files:
        context.append(get_file_summary(f))

# 5. Embedding-based retrieval (Cursor's secret sauce)
    similar_chunks = vector_search(
        query=current_code_context,
        index=codebase_embeddings,
        top_k=10
    )
    context.extend(similar_chunks)

return truncate_to_budget(context, max_tokens=8000)

Step 3: Embedding-based retrieval — Cursor indexes your entire codebase as embeddings. When you're writing code, it searches for semantically similar code chunks — even if they're in files you haven't opened.

This is essentially RAG for code (Episode 22), and it's why Cursor feels like it "understands" your project.

🔧

The Completion Pipeline

Here's what happens in the ~200ms between you pausing and seeing a suggestion:

1. Trigger detected (you paused typing for ~300ms)

│

1. Context gathered (current file + LSP + embeddings) ~50ms

│

1. FIM prompt constructed

│ [context + code before cursor] │ [code after cursor] │

│

1. Sent to model (local or API) ~100ms

│ - Copilot: GitHub's cloud (Codex/GPT-4)

│ - Cursor: Mix of cloud + local models

│ - Codeium: Their optimized inference

│

1. Response tokens generated (usually 1-10 lines) ~50ms

│

1. Post-processing ~10ms

│ - Validate syntax (AST parsing)

│ - Check bracket matching

│ - Apply indentation rules

│ - Trim to logical boundary (don't stop mid-expression)

│

1. Ghost text rendered in editor

Total time: ~200-400ms — fast enough to feel instantaneous after a natural typing pause.

Why Speed Matters

Completion needs to be FAST. If it takes more than 500ms, it arrives after you've already started typing the next character, and it's useless.

This is why coding assistants use:

Smaller, faster models for basic completions (not GPT-4)

Speculative decoding to speed up generation

Aggressive caching — if the context hasn't changed much, reuse the previous KV cache

Cancellation — if you start typing, cancel the in-flight request immediately

📊

Copilot vs Cursor: Architecture Differences

Feature GitHub Copilot Cursor

Cursor's key innovation: The Composer feature. Instead of completing one line at a time, you describe a change in natural language, and it edits multiple files simultaneously:

User: "Add error handling to all API routes and log errors to Sentry"

Cursor Composer:

├── Modifies routes/users.js (adds try-catch)

├── Modifies routes/orders.js (adds try-catch)

├── Modifies routes/products.js (adds try-catch)

├── Creates lib/error-handler.js (shared error handling)

└── Updates package.json (adds @sentry/node)

This is fundamentally different from completion — it's code transformation at project scale.

The Tab-Tab-Tab Flow: How Predictions Chain

Modern coding assistants don't just predict the next line — they predict your next action:

Step 1: You type "def validate_email("

→ Suggestion: "email: str) -> bool:"

→ Tab ✓

Step 2: Cursor now in function body

→ Suggestion: "pattern = r'^[\w\.-]+@[\w\.-]+\.\w+$'"

→ Tab ✓

Step 3: Next line

→ Suggestion: "return bool(re.match(pattern, email))"

→ Tab ✓

Step 4: After function

→ Suggestion: "def validate_phone(phone: str) -> bool:"

→ Tab ✓ (it predicted you'd write a similar validator!)

This "Tab flow" is addictive because the model isn't just completing code — it's predicting your intent. After seeing you write a validation function, it anticipates you'll write more validation functions.

Under the Hood: The Models

GitHub Copilot

Originally used OpenAI Codex (a GPT-3 fine-tuned on code). Now uses a mix:

Fast completions: Custom small model (~1-3B params)

Complex completions: GPT-4-mini

Chat: GPT-4o

Cursor

Uses a mix of models:

Fast completions: Custom model (rumored ~3B)

Complex suggestions: Claude 3.5 Sonnet or GPT-4

Composer: Claude 3.5 Sonnet (excels at multi-file edits)

Embeddings: Custom embedding model for codebase indexing

Code-Specific Training

These models are fine-tuned on massive code datasets:

The Stack — 6.4TB of permissively licensed code from GitHub

StarCoder training data — 1T tokens of code in 86 languages

Code review data — PR comments, code reviews, bug fixes

Documentation — READMEs, API docs, StackOverflow

The fine-tuning isn't just "learn to write code." It includes:

Fill-in-the-middle training (the FIM objective)

Instruction following for code tasks

Bug detection and fix generation

Code explanation and documentation

🔬

The Latency-Quality Tradeoff

Every coding assistant makes this tradeoff constantly:

Fast but dumb (30ms): Complete the current line

Low accuracy, high speed
                          Use: small local model

Medium (200ms):           Complete the current block
Good accuracy, acceptable speed
                          Use: optimized cloud model

Slow but smart (2-5s): Rewrite entire function, multi-file

High accuracy, user-initiated
                          Use: GPT-4 / Claude 3.5 Sonnet

The UI adapts: inline ghost text for fast suggestions, a chat panel for slow-but-smart answers, and Composer for project-wide changes.

🛡️

Privacy and Security

A real concern: your code is being sent to external servers.

Feature Copilot Business Cursor Codeium

For sensitive code, options include:

Copilot Enterprise — self-hosted, on-prem

Local models — StarCoder 2, CodeLlama, DeepSeek Coder running locally

Codeium self-hosted — run their inference on your infra

Local models are 3-5x worse than cloud models. That's the tradeoff.

📦

Practical Takeaways

Fill-in-the-middle is the key innovation — knowing what comes AFTER the cursor is as important as what comes before

Context gathering is 80% of the magic — LSP integration, codebase embeddings, and recent edits determine suggestion quality

Speed matters more than perfection — a 200ms decent suggestion beats a 2-second perfect one for inline completions

Cursor's Composer is a paradigm shift — multi-file edits from natural language descriptions change how you code

The "Tab flow" is intentional — models predict your intent, not just the next token

Privacy tradeoffs are real — cloud models are better, but your code leaves your machine

🚀

What's Next?

Episode 94: How Perplexity Works — Another real-world architecture, completely different from coding assistants. Perplexity combines search, RAG, and citations into an "answer engine." We'll trace how a question goes from your keyboard through web crawling, retrieval, and generation — with citations that actually point to real sources.

← Previous

Ep 92: How ChatGPT Works

Next →

Ep 94: How Perplexity Works

Next: Episode 94 — How Perplexity Works

Google gives you 10 blue links. Perplexity gives you the answer with citations. Here's the architecture behind it.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.