Monitoring & Observability: Seeing Your AI in Production

Your AI app works on your laptop. You tested it. The demos were great.

Then you deployed it. Real users hit it. Edge cases emerged. Someone pasted a 10,000-word document. Another user typed "ignore all previous instructions and..."

Without monitoring, you're flying blind. And flying blind with AI is expensive, risky, and eventually embarrassing.

This episode covers how to observe AI systems in production — what to track, how to track it, and which tools make it painless.

Why AI Observability Is Different

Traditional app monitoring: track latency, errors, throughput. Done.

AI observability adds a new dimension: the quality of the output.

Traditional Monitoring AI Observability

HTTP 200 = success HTTP 200 but response is hallucinated

The Three Pillars: Logs, Traces, Metrics

1. Logging: What Happened

Log everything you might need to debug:

python

import json

import time

from datetime import datetime

class LLMLogger:

def log_interaction(self, request, response, metadata):
        log_entry = {
            "timestamp": datetime.utcnow().isoformat(),
            "request_id": metadata["request_id"],
            "model": metadata["model"],
            "prompt_preview": request["prompt"][:200],  # Truncate
            "response_preview": response["text"][:200],
            "tokens": {
                "input": response["usage"]["prompt_tokens"],
                "output": response["usage"]["completion_tokens"],
                "total": response["usage"]["total_tokens"]
            },
            "latency_ms": metadata["latency_ms"],
            "user_id": metadata.get("user_id"),
            "session_id": metadata.get("session_id"),
            "finish_reason": response.get("finish_reason")  # length, stop, etc.
        }

# Send to your log aggregator
        self.send_to_elasticsearch(log_entry)

Critical fields to log:

Prompt/response (truncated for PII)

Token counts

Latency breakdown (TTFT, total)

Model version

Error details

User/session IDs for tracing

1. Tracing: Follow the Request

A single user request might hit multiple services:

User Request → Auth Service → Rate Limiter → Cache Check → LLM API → Post-processing → Response

Distributed tracing connects these spans:

python

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

@tracer.start_as_current_span("llm_request")

def call_llm(prompt: str):

with tracer.start_as_current_span("cache_check"):
        cached = check_cache(prompt)
        if cached:
            return cached

with tracer.start_as_current_span("api_call"):
        response = openai.chat.completions.create(...)

with tracer.start_as_current_span("post_process"):
        result = clean_output(response)

return result

What tracing reveals:

Which step is slowest

Where failures originate

How components interact

1. Metrics: Aggregate Patterns

Track aggregates, not individual events:

Metric Why It Matters Alert Threshold

🔧

Eval in Production: Is the Output Good?

This is the hard part. How do you know if your AI's responses are actually good?

Automated Evaluation

1. Structured output validation

python

def validate_json_output(response: str, schema: dict) -> bool:
try:
        data = json.loads(response)
        jsonschema.validate(data, schema)
        return True
    except (json.JSONDecodeError, ValidationError):
        return False

1. Hallucination detection

python

def check_hallucination(response: str, context: str) -> float:
"""Use embeddings to check if response is grounded in context"""
    response_embedding = embed(response)
    context_embedding = embed(context)
    similarity = cosine_similarity(response_embedding, context_embedding)
    return similarity  # Low similarity = possible hallucination

1. Sentiment drift

Track if your AI's tone changes over time:

python

def analyze_sentiment(text: str) -> dict:
# Use a sentiment classifier
    return {"positive": 0.7, "neutral": 0.2, "negative": 0.1}

Alert if sentiment distribution shifts significantly

📊

Tools of the Trade

Managed Solutions

Tool Best For Pricing

LangSmith Example

python

from langsmith import Client

from langchain.callbacks.tracers import LangChainTracer

tracer = LangChainTracer()

Your chain automatically logs to LangSmith

chain = promptllmoutput_parser

result = chain.invoke({"question": user_input}, callbacks=[tracer])

View full trace: prompt → LLM call → output → parsing

Helicone Example

python

Just change the base URL, keep everything else

import openai

openai.api_base = "https://oai.helicone.ai/v1"

openai.default_headers = {

"Helicone-Auth": f"Bearer {HELI_CONE_API_KEY}",
    "Helicone-User-Id": user_id,
    "Helicone-Session-Id": session_id

}

All requests automatically tracked with cost, latency, metadata

Custom Dashboards: Build Your Own

Sometimes you need specific metrics. Here's a Grafana setup:

json

{

"dashboard": {

"panels": [
      {
        "title": "Requests Per Minute",
        "targets": [{
          "expr": "rate(llm_requests_total[5m])"
        }]
      },
      {
        "title": "P95 Latency by Model",
        "targets": [{
          "expr": "histogram_quantile(0.95, rate(llm_latency_bucket[5m]))"
        }]
      },
      {
        "title": "Cost Per Hour",
        "targets": [{
          "expr": "sum(llm_cost_dollars)"
        }]
      }
    ]

}

}

Alerting: Tell Me When Things Break

Critical Alerts (Page Me)

Error rate > 5%

P99 latency > 10 seconds

Cost per hour 3x baseline

Context window exhaustion events

Warning Alerts (Check Tomorrow)

Cache hit rate dropping

Token usage trending up

New error types appearing

Model version deprecated notice

Setup Example (PagerDuty)

yaml

alerts:

• name: high_error_rate

condition: error_rate > 0.05
    duration: 5m
    severity: critical

• name: latency_degradation

condition: p99_latency > 5000ms
    duration: 10m
    severity: warning

🔬

Practical Takeaways

Log structured data — JSON logs you can query, not text dumps

Use distributed tracing — Follow requests across services

Track AI-specific metrics — Token usage, context utilization, not just latency

Implement automated evaluation — Detect hallucinations, validate outputs

Start with managed tools — LangSmith or Helicone before building custom

Set meaningful alerts — Not just "errors > 0", but patterns that matter

Monitor costs per user — Find your expensive users, understand why

🛡️

What's Next?

Episode 83: Semantic Caching — Cache by meaning, not exact match. Embedding similarity, GPTCache, and the 40-70% cost savings you're probably missing.

← Previous

Ep 55: Cost Optimization

Next →

Ep 57: Mixture of Experts

Next: Episode 57 — Mixture of Experts

What if GPT-4 doesn't use all its parameters for every question? It doesn't. That's the Mixture of Experts trick.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.