How Voice Assistants Work: STT → LLM → TTS Pipeline

"Hey Siri, what's the weather?"

Two seconds later: "It's 32 degrees and humid in Mumbai."

That two-second journey is a miracle of engineering. Your voice becomes text. Text becomes understanding. Understanding becomes speech. All while you're holding a ₹50,000 computer in your hand that runs on a battery.

Voice assistants aren't magic. They're a pipeline. And each stage has its own challenges, latency budgets, and failure modes.

Let's trace the journey from sound wave to spoken response.

The Pipeline Overview

Voice Input → Wake Word → ASR/STT → NLU → LLM → TTS → Audio Output

(you) (local) (cloud) (cloud) (cloud) (cloud) (speaker)

Stage What Happens Typical Latency

Stage 1: Wake Word Detection

Your device is always listening. But it can't send everything to the cloud — bandwidth, privacy, battery.

So it runs a tiny model locally:

python

Simplified wake word detection

import tensorflow as tf

Model is <1MB, runs on device CPU

interpreter = tf.lite.Interpreter(model_path="hey_siri.tflite")

interpreter.allocate_tensors()

def process_audio_chunk(audio_chunk):

# MFCC feature extraction
    features = extract_mfcc(audio_chunk)

# Run inference
    interpreter.set_tensor(input_index, features)
    interpreter.invoke()
    prediction = interpreter.get_tensor(output_index)

# If confidence > threshold, wake word detected
    return prediction[0] > 0.8

Key Constraints

Constraint Why It Matters

Popular wake word engines: Picovoice Porcupine, Snowboy, TensorFlow Lite.

🔧

Stage 2: Automatic Speech Recognition (ASR/STT)

Once the wake word triggers, the real work begins. Your speech needs to become text.

OpenAI Whisper

The current gold standard:

python

import whisper

model = whisper.load_model("base")  # tiny, base, small, medium, large
result = model.transcribe("audio.mp3")
print(result["text"])  # "What's the weather like today?"

How Whisper Works

Audio (30s chunks) → Log-Mel Spectrogram → Encoder → Decoder → Text Tokens

Model Parameters Speed Word Error Rate

Streaming ASR

For real-time assistants, you can't wait for the user to finish speaking:

python

Streaming transcription

import speech_recognition as sr

recognizer = sr.Recognizer()
microphone = sr.Microphone()

def callback(recognizer, audio):

try:
        # Send chunk to ASR API
        text = recognizer.recognize_whisper(audio, model="base")
        print(f"Heard: {text}")
    except sr.UnknownValueError:
        pass

Background listening

stop_listening = recognizer.listen_in_background(microphone, callback)

📊

Stage 3: Natural Language Understanding

Now we have text. What does the user want?

"What's the weather like today?"

↓

Intent: weather_inquiry

Entities: { "time": "today", "location": "current" }

Intent Classification

python

Simple intent classifier

INTENTS = ["weather", "timer", "reminder", "music", "general_knowledge"]
def classify_intent(text: str) -> str:
# Can use small model or even regex for simple cases
    if any(word in text for word in ["weather", "temperature", "rain"]):
        return "weather"
    elif any(word in text for word in ["timer", "countdown"]):
        return "timer"
    # ... fallback to LLM
    return "general"

Modern Approach: Just Use an LLM

With modern LLMs, you can skip explicit NLU:

python

response = openai.chat.completions.create(
model="gpt-4",
    messages=[
        {"role": "system", "content": "You are a helpful voice assistant."},
        {"role": "user", "content": user_text}
    ]

)

The LLM handles intent, entities, and response generation in one shot.

Stage 4: Large Language Model

This is the brain. See Episode 4 for how inference works.

For voice assistants, LLM selection is a latency vs quality tradeoff:

Model Latency Quality Use Case

GPT-4 2-3s Highest Complex queries

GPT-3.5-turbo 0.5-1s High General purpose

Claude Haiku 0.3-0.5s Good Simple queries

Self-hosted 7B 0.2-0.5s Moderate Privacy-critical

Streaming Responses

Voice assistants can't wait for the full response:

python

Stream tokens as they're generated

stream = openai.chat.completions.create(
model="gpt-3.5-turbo",
    messages=messages,
    stream=True  # Enable streaming

)

for chunk in stream:

if chunk.choices[0].delta.content:
        text_chunk = chunk.choices[0].delta.content
        # Send to TTS immediately
        tts_engine.synthesize(text_chunk)

Stage 5: Text-to-Speech (TTS)

The final step: turn text into natural-sounding speech.

Types of TTS

Type How It Works Quality Speed

Concatenative Stitch recorded audio fragments Robotic Fast

Parametric Generate audio from parameters Better Fast

Neural Deep learning end-to-end Human-like Slower

Modern Neural TTS

python

from TTS.api import TTS

Load model

tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2")

Clone voice from sample

tts.tts_to_file(

text="Hello, this is a test of neural text to speech.",
    speaker_wav="reference_voice.wav",
    language="en",
    file_path="output.wav"

)

Popular TTS Services

Service Quality Latency Cost

🔬

The Latency Challenge

Users expect sub-second responses. Here's how to hit that:

Optimization Strategies

1. Pipeline Parallelism

ASR chunk 1 → ASR chunk 2 → ASR chunk 3

↓           ↓           ↓
     LLM start → LLM cont → LLM finish
       ↓           ↓           ↓
     TTS start → TTS cont → Audio play

Start TTS on partial LLM output, don't wait for completion.

1. Speculative TTS

Pre-generate common responses:

"I'm not sure I understand"

"Let me think about that"

"Here's what I found"

1. Edge Deployment

python

Run small models locally

Whisper tiny (39M) → GPT-2 (124M) → Piper TTS (local)

Total latency: <500ms, zero cloud cost

🛡️

Try It Yourself

Build a simple voice assistant in Python:

bash

pip install openai whisper TTS pyaudio

python

import whisper

import openai

from TTS.api import TTS

import speech_recognition as sr

class VoiceAssistant:

def __init__(self):
        self.asr = whisper.load_model("base")
        self.tts = TTS("tts_models/en/ljspeech/tacotron2-DDC")

def listen(self):
        r = sr.Recognizer()
        with sr.Microphone() as source:
            print("Listening...")
            audio = r.listen(source)
        return audio

def transcribe(self, audio):
        result = self.asr.transcribe(audio)
        return result["text"]

def think(self, text):
        response = openai.chat.completions.create(
            model="gpt-3.5-turbo",
            messages=[{"role": "user", "content": text}]
        )
        return response.choices[0].message.content

def speak(self, text):
        self.tts.tts_to_file(text=text, file_path="response.wav")
        # Play audio...

def run(self):
        audio = self.listen()
        text = self.transcribe(audio)
        response = self.think(text)
        self.speak(response)

📦

Practical Takeaways

Voice = pipeline of 5 stages — Each adds latency, each can fail

Wake word runs locally — Privacy and battery require tiny on-device models

Whisper is the ASR standard — Open source, accurate, multilingual

LLMs can replace NLU — Intent classification + entity extraction in one model

Streaming is essential — Don't wait for full generation, pipeline the stages

Neural TTS is indistinguishable from human — ElevenLabs, OpenAI lead here

Sub-second response requires optimization — Edge deployment, caching, speculation

🚀

What's Next?

Episode 97: How Recommendation Systems Use AI — Netflix, YouTube, Spotify. Collaborative filtering, embeddings, and the AI that decides what you watch next.

← Previous

Ep 95: How Midjourney Works

Next →

Ep 97: How Recommendation Systems Use AI

Next: Episode 97 — How Recommendation Systems Use AI

You open Netflix. The perfect show is already on your home screen. That's not luck — it's a recommendation engine powered by AI.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.