Structured Output / JSON Mode: Making AI Responses Machine-Readable

You ask the model to return JSON. It returns:

Sure! Here's the JSON:

{"name": "Abhishek", "age": 30}
Hope that helps!


Your `JSON.parse()` explodes. The model wrapped valid JSON inside natural language and a markdown code block. This happens constantly — and it breaks every pipeline that tries to extract structured data from LLM responses.

**Structured output** solves this. It guarantees the model's response is valid JSON. Not "probably JSON." Not "JSON with some extra text." **Guaranteed valid JSON that matches your schema.** Every single time.

How? By constraining the model's token generation at the decoding level — literally making it impossible to output invalid tokens.

---

## The Three Levels of Structure

### Level 1: Prompt Engineering (Unreliable)

response = client.chat.completions.create(
model="gpt-4o",
    messages=[{
        "role": "user",
        "content": "Extract the name and age from: 'Abhishek is 30 years old.' Return ONLY valid JSON, nothing else."
    }]

)

Sometimes works, sometimes adds "Here's the JSON:", sometimes uses markdown

Success rate: ~70-85%. Fine for demos. Terrible for production.

Level 2: JSON Mode (Guaranteed JSON, No Schema)

python

response = client.chat.completions.create(
model="gpt-4o",
    messages=[{
        "role": "system",
        "content": "Return valid JSON with 'name' and 'age' fields."
    }, {
        "role": "user",
        "content": "Abhishek is 30 years old."
    }],
    response_format={"type": "json_object"}  # ← JSON mode

)

result = json.loads(response.choices[0].message.content)

Always valid JSON, but schema not guaranteed

Could return {"name": "Abhishek", "age": 30}

Or {"person": "Abhishek", "years": "30"} — valid JSON, wrong schema

Success rate: 100% valid JSON, ~90% correct schema.

Level 3: Strict Structured Output (Guaranteed Schema)

python

response = client.chat.completions.create(
model="gpt-4o",
    messages=[{
        "role": "user",
        "content": "Abhishek is 30 years old."
    }],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "person_info",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "name": {"type": "string"},
                    "age": {"type": "integer"}
                },
                "required": ["name", "age"],
                "additionalProperties": False
            }
        }
    }

)

result = json.loads(response.choices[0].message.content)

ALWAYS: {"name": "Abhishek", "age": 30}

Schema guaranteed. Types guaranteed. Required fields guaranteed.

Success rate: 100% valid JSON, 100% matching schema.

How Constrained Generation Works

The magic behind guaranteed structured output is constrained decoding — modifying the token probabilities at generation time to make invalid tokens impossible.

The Token Probability Approach

At each step, the model produces a probability distribution over all ~100,000 tokens. Normally, any token can be selected. Constrained decoding masks tokens that would violate the schema:

Normal generation:

Token probabilities: {"Hello": 0.3, "{": 0.1, "Sure": 0.2, ...}

Selected: "Hello" (or any token)

JSON mode generation:

Token probabilities: {"Hello": 0.3, "{": 0.1, "Sure": 0.2, ...}

Valid tokens for position 0: only "{"

Masked: {"Hello": 0, "{": 1.0, "Sure": 0, ...}

Selected: "{" (guaranteed)

After {, the next valid tokens are " (start of key) or } (empty object). After "name", the next valid token must be ":. And so on.

Finite State Machine (FSM) Approach

The schema is compiled into a finite state machine (FSM) that tracks which tokens are valid at each step:

State 0: Expect "{"

State 1: Expect "\"name\"" or "\"age\""

State 2: After "\"name\"", expect ":"

State 3: After ":", expect string token

State 4: Expect "," or "}"

...

The FSM pre-computes valid token sets for each state. At each generation step, only valid tokens get non-zero probability.

python

Simplified FSM-based constrained generation

class JSONConstraint:

def __init__(self, schema):
        self.fsm = compile_schema_to_fsm(schema)
        self.state = self.fsm.initial_state

def get_valid_tokens(self):
        """Return set of token IDs valid in current state."""
        return self.fsm.valid_tokens(self.state)

def advance(self, token_id):
        """Update state after generating a token."""
        self.state = self.fsm.transition(self.state, token_id)

During generation:

constraint = JSONConstraint(schema)

for step in range(max_tokens):

logits = model.forward(input_ids)  # Raw token probabilities

valid_tokens = constraint.get_valid_tokens()
    logits[~valid_tokens] = float('-inf')  # Mask invalid tokens

token_id = sample(logits)
    constraint.advance(token_id)

Outlines: Open-Source Constrained Generation

Outlines is the leading open-source library for structured generation. It works with any model (Hugging Face, vLLM, llama.cpp):

python

import outlines

model = outlines.models.transformers("meta-llama/Llama-3-8B-Instruct")

Generate valid JSON matching a Pydantic schema

from pydantic import BaseModel

class Person(BaseModel):

name: str
    age: int
    occupation: str
    skills: list[str]

generator = outlines.generate.json(model, Person)
result = generator("Extract person info: Abhishek is a 30-year-old AI engineer who knows Python and Rust")
print(result)
Person(name='Abhishek', age=30, occupation='AI engineer', skills=['Python', 'Rust'])

print(type(result))

— it's a Pydantic object, fully typed!

Outlines: Beyond JSON

Outlines can constrain to any regular expression or context-free grammar:

python

Regex-based generation

import outlines

model = outlines.models.transformers("meta-llama/Llama-3-8B-Instruct")

Generate valid email addresses

email_gen = outlines.generate.regex(model, r'[a-z]+@[a-z]+\.[a-z]{2,4}')
email = email_gen("Generate an email for Abhishek")

"abhishek@example.com" — guaranteed valid email format

Generate valid dates

date_gen = outlines.generate.regex(model, r'\d{4}-\d{2}-\d{2}')
date = date_gen("When was the transformer paper published?")

"2017-06-12" — guaranteed YYYY-MM-DD format

Generate valid Python

python_gen = outlines.generate.cfg(model, "python")  # Uses Python grammar
code = python_gen("Write a fibonacci function")

Guaranteed syntactically valid Python

🔧

Grammar-Based Generation: How It Works Under the Hood

The most powerful approach compiles a formal grammar (like JSON schema, regex, or BNF) into token-level constraints:

Step 1: Schema → Grammar

JSON Schema: {"type": "object", "properties": {"name": {"type": "string"}, "age": {"type": "integer"}}}

Converts to grammar:

root → "{" members "}"

members → pair ("," pair)*

pair → string ":" value

string → '"' chars '"'

value → stringinteger

integer → digit+

Step 2: Grammar → Token Mask

At position 0: grammar expects "{"

Valid tokens: ["{"] → mask everything else

At position 1: grammar expects '"'

Valid tokens: ['"'] → mask everything else

At position 2: grammar expects key name

Valid tokens: any string character tokens

(But schema says keys must be "name" or "age")

Valid tokens: ["n", "a"] → mask everything else

Step 3: Generate with Mask

Each token is sampled from the model's distribution, but only valid tokens can be selected. The model's "creativity" still determines which valid option to pick — the grammar just prevents invalid ones.

📊

Provider Comparison

Provider JSON Mode Strict Schema Method

vLLM with Structured Output

python

from vllm import LLM, SamplingParams
llm = LLM(model="meta-llama/Llama-3-8B-Instruct")
params = SamplingParams(
temperature=0.7,
    max_tokens=256,
    guided_json={  # Outlines-powered constrained generation
        "type": "object",
        "properties": {
            "sentiment": {"type": "string", "enum": ["positive", "negative", "neutral"]},
            "confidence": {"type": "number"},
            "keywords": {"type": "array", "items": {"type": "string"}}
        },
        "required": ["sentiment", "confidence", "keywords"]
    }

)

outputs = llm.generate(["Analyze sentiment: 'This product is amazing!'"], params)

Guaranteed valid JSON matching the schema

llama.cpp with GBNF Grammar

bash

GBNF grammar for structured output

cat > sentiment.gbnf << 'EOF'

root   ::= "{" ws "\"sentiment\":" ws sentiment "," ws "\"score\":" ws number "}"
sentiment ::= "\"positive\"""\"negative\"""\"neutral\""
number ::= [0-9] "." [0-9]+
ws     ::= [ \t\n]*

EOF

Generate with grammar constraint

./llama-cli -m llama-3-8b.gguf \

--grammar-file sentiment.gbnf \
    -p "Analyze: 'This movie was terrible'"

Output: {"sentiment":"negative","score":0.92}

Performance Impact

Constrained generation isn't free — there's a small overhead:

Metric Unconstrained JSON Mode Strict Schema Overhead

The 2-5% speed overhead is negligible compared to the benefit of never parsing invalid output.

The Real Cost: Token Efficiency

Constrained generation can actually save tokens because the model doesn't waste tokens on preambles like "Sure, here's the JSON:":

Unconstrained: "Sure! Here's the extracted data:\n\n``json\n{\"name\": ...}\n``" = 45 tokens

Constrained: "{\"name\": ...}" = 15 tokens

3x fewer output tokens = 3x cheaper

Try It Yourself

python

Compare all three approaches

from openai import OpenAI

import json

client = OpenAI()
prompt = "Extract: 'The Godfather (1972) directed by Francis Ford Coppola, rated 9.2/10'"

Level 1: Prompt engineering

r1 = client.chat.completions.create(
model="gpt-4o-mini",
    messages=[{"role": "user", "content": f"Return JSON only. {prompt}"}]

)

print("Prompt engineering:", r1.choices[0].message.content[:100])

Level 2: JSON mode

r2 = client.chat.completions.create(
model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Return JSON with title, year, director, rating fields."},
        {"role": "user", "content": prompt}
    ],
    response_format={"type": "json_object"}

)

print("JSON mode:", json.loads(r2.choices[0].message.content))

Level 3: Strict schema

r3 = client.chat.completions.create(
model="gpt-4o-mini",
    messages=[{"role": "user", "content": prompt}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "movie",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "title": {"type": "string"},
                    "year": {"type": "integer"},
                    "director": {"type": "string"},
                    "rating": {"type": "number"}
                },
                "required": ["title", "year", "director", "rating"],
                "additionalProperties": False
            }
        }
    }

)

print("Strict schema:", json.loads(r3.choices[0].message.content))

{"title": "The Godfather", "year": 1972, "director": "Francis Ford Coppola", "rating": 9.2}

🔬

Practical Takeaways

Prompt engineering for JSON is unreliable — don't use it in production pipelines

JSON mode guarantees valid JSON but not schema — use when you just need parseable output

Strict schema mode guarantees both JSON and schema — use for production data extraction

Constrained generation works by masking invalid tokens — FSM-based approach, 2-5% overhead

Outlines is the go-to open-source library — works with any model, supports JSON, regex, and full grammars

Structured output saves tokens — no preambles, no code blocks, just the data

🛡️

What's Next?

Episode 81: A2A (Agent-to-Agent Protocol) — MCP connects models to tools. But what about models talking to other models? Google's A2A protocol defines how autonomous agents discover each other, negotiate capabilities, and collaborate on tasks. It's the next layer of the AI stack.

← Previous

Ep 79: OpenAI Function Calling Spec

Next →

Ep 81: A2A (Agent-to-Agent)

Next: Episode 81 — A2A (Agent-to-Agent)

MCP lets AI talk to tools. Function calling lets AI take actions. But what happens when you need AI talking to AI?

This is part of a 98-episode series covering AI engineering from tokens to production deployment.