Voice & TTS: How AI Turns Text Into Human Speech

Close your eyes and listen to someone speak. Not the words — the way they speak. The pause before a punchline. The rising pitch of a question. The warmth when they say your name. The way their voice cracks when they're tired.

Now try to build a machine that does all of that from raw text.

That's Text-to-Speech (TTS) — and it's gone from robotic "PLEASE. LEAVE. A. MESSAGE." to voices so real you genuinely can't tell the difference. ElevenLabs can clone your voice from 30 seconds of audio. OpenAI's real-time voice can hold a conversation with emotional inflection. CosyVoice can generate speech in languages the speaker never spoke.

Here's how it all works.

The Two Sides: STT and TTS

Voice AI has two directions:

Direction Full Name What It Does Key Models

Both are hard, but for completely different reasons. STT needs to handle accents, background noise, multiple speakers, and domain-specific vocabulary. TTS needs to sound human — with proper rhythm, emotion, and naturalness.

Speech-to-Text: How Whisper Works

Whisper (OpenAI, 2022) is the most widely used STT model. It's open-source, supports 99 languages, and handles noisy audio remarkably well.

Architecture

Whisper is an encoder-decoder transformer — similar to the original transformer from the "Attention Is All You Need" paper:

Audio Waveform

↓

Mel Spectrogram (80 channels × time)

↓

Convolutional Stem (2 conv layers)

↓

Transformer Encoder (process audio features)

↓

Transformer Decoder (generate text tokens autoregressively)

↓

Text Output

Step 1: Audio to Spectrogram

Raw audio is a 1D waveform — just amplitude over time. Whisper converts it to a Mel spectrogram — a 2D representation that shows frequency content over time:

Raw audio: [0.1, -0.3, 0.5, -0.2, ...] (16kHz = 16,000 samples/sec)

↓

Mel spectrogram: 80 frequency bins × T time frames

(basically a "heatmap" of sound frequencies)

Think of it like sheet music — the y-axis shows pitch (frequency), the x-axis shows time, and the intensity shows volume. This representation makes it much easier for the model to understand speech.

Step 2: Encode

The encoder processes 30-second audio chunks. Two convolutional layers first downsample the spectrogram, then transformer layers process the sequence:

Input: 80 × 3000 (80 Mel bins × 30 seconds at 100 frames/sec)

After conv: 1500 × d_model

After encoder: 1500 × d_model (contextualized audio features)

Step 3: Decode

The decoder generates text tokens one at a time, attending to both previous text tokens and the encoder's audio features. It uses cross-attention — the decoder's queries attend to the encoder's keys and values:

Decoder input: [, , ] Decoder output: "Hello", "how", "are", "you", "today",

Special tokens control behavior:

— language tag — transcription mode (vs for translation) — whether to include timestamps

Whisper Model Sizes

Model Params VRAM Speed (relative) English WER

WER = Word Error Rate. Whisper large-v3 gets 97.8% of words correct on English speech — better than many human transcribers.

🔧

Text-to-Speech: The Modern Pipeline

TTS has evolved through three generations:

Generation 1: Concatenative (1990s–2010s)

Record a voice actor saying thousands of phrases. Chop the recordings into phoneme snippets. Stitch them together for new sentences. Sounded robotic because the stitching was never seamless.

Generation 2: Neural TTS (2016–2022)

Use neural networks to generate speech from scratch. Tacotron (Google, 2017) and WaveNet (DeepMind, 2016) were breakthroughs — they generated mel spectrograms from text, then converted spectrograms to audio waveforms.

Text → Tacotron (text → mel spectrogram) → WaveNet (mel → audio waveform)

Generation 3: End-to-End Neural (2023+)

Modern TTS models generate audio directly from text using transformer or diffusion architectures, often producing speech indistinguishable from real humans.

📊

How Modern TTS Works

Let's break down the pipeline of a state-of-the-art TTS system:

Step 1: Text Processing

Raw text needs normalization before the model sees it:

Input: "Dr. Smith earned $1.5M in Q3 2024"

Normalized: "Doctor Smith earned one point five million dollars in queue three twenty twenty four"

This includes:

Abbreviation expansion (Dr. → Doctor)

Number verbalization ($1.5M → one point five million dollars)

Punctuation to prosody cues (commas → short pauses)

Step 2: Phoneme Conversion

Text gets converted to phonemes — the actual sounds of speech:

"Hello world" → /h ɛ l oʊ w ɜː r l d/

This handles the irregularity of English spelling. "Through," "though," "thought," "tough" — four different pronunciations from similar spellings. A grapheme-to-phoneme (G2P) model handles this conversion.

Step 3: Acoustic Model

The core neural network that converts phonemes (+ speaker embedding + emotion embedding) into an audio representation:

Inputs:

• Phoneme sequence: [h, ɛ, l, oʊ, ...]
• Speaker embedding: [0.23, -0.45, ...] (voice identity)
• Style/emotion: [neutral, happy, sad, ...]
• Prosody features: [pitch contour, duration, energy]

Output:

• Mel spectrogram: 80 channels × T frames
• OR directly: audio codec tokens

Step 4: Vocoder (Spectrogram → Audio)

The vocoder converts the mel spectrogram into an actual audio waveform. Modern vocoders include:

Vocoder Speed Quality Notes

HiFi-GAN is the most popular — it generates audio faster than real-time with near-perfect quality.

Voice Cloning: How ElevenLabs Does It

Voice cloning creates a speaker embedding — a vector that captures everything unique about a voice: pitch, timbre, accent, speaking style, breathiness.

The Process

Reference audio (30 sec) → Speaker Encoder → Speaker Embedding [256-dim vector]

↓

New text → TTS Model + Speaker Embedding → Cloned speech

The speaker encoder is trained on thousands of voices to extract a compact representation of vocal identity. Once you have this embedding, you can generate any text in that voice.

What the Embedding Captures

Dimension What It Encodes

Pitch range How high/low the voice naturally sits

With just 30 seconds of clean audio, modern systems can create a remarkably accurate voice clone. ElevenLabs' "Instant Voice Cloning" works with as little as 1 minute; their "Professional Voice Cloning" uses 30+ minutes for studio-quality results.

Real-time Voice: GPT-4o's Approach

GPT-4o's voice mode doesn't use a traditional TTS pipeline. Instead, it generates audio tokens directly:

Traditional: Text → LLM → Text response → TTS → Audio

GPT-4o: Audio → LLM → Audio tokens → Audio decoder → Audio

The LLM itself predicts audio tokens — discrete codes from a neural audio codec like EnCodec (Meta). This means:

No text intermediate — the model goes directly from understanding to speaking

Emotional expression — the model can express laughter, hesitation, excitement without explicit programming

Low latency — no waiting for a separate TTS pipeline

Interruption handling — the model can stop mid-sentence and respond to new input

This is why GPT-4o's voice sounds so natural compared to "read text aloud" approaches.

🔬

Neural Audio Codecs: The Foundation

Modern voice AI relies on neural audio codecs — models that compress audio into discrete tokens:

EnCodec (Meta)

Audio waveform → Encoder → Residual Vector Quantization → Discrete tokens

↓
                           Decoder ← ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─┘
                              ↓
                        Reconstructed audio

EnCodec compresses audio to as low as 1.5 kbps — that's 10x more compressed than MP3 — while maintaining good speech quality. The discrete tokens are what LLMs learn to predict.

Residual Vector Quantization (RVQ): Instead of one codebook, EnCodec uses 8 codebooks in layers. The first captures coarse structure (pitch, rhythm), each subsequent one adds finer details (timbre, harmonics). It's like painting in layers — rough sketch first, then details.

🛡️

CosyVoice: Open-Source Voice Cloning

CosyVoice (Alibaba, 2024) is an open-source TTS model that supports:

Zero-shot voice cloning — clone from a few seconds of audio

Cross-lingual synthesis — speak a language the reference speaker never spoke

Emotion control — happy, sad, angry, whispering

Fine-grained control — adjust speed, pitch, and style

bash

CosyVoice usage (Python)

from cosyvoice import CosyVoice

model = CosyVoice('CosyVoice-300M-Instruct')

Zero-shot voice cloning

output = model.inference_zero_shot(
text="Welcome to Learn AI Infrastructure",
    prompt_text="reference transcript",
    prompt_speech="reference_audio.wav"

)

Cross-lingual (speak Hindi with English voice)

output = model.inference_cross_lingual(
text="नमस्ते, मैं एक AI हूं",
    prompt_speech="english_speaker.wav"

)

📦

Edge TTS: Practical and Free

For practical use, Edge TTS (Microsoft's neural voices via the Edge browser API) offers high-quality synthesis for free:

bash

Install

pip install edge-tts

Basic usage

edge-tts --text "Hello from Learn AI Infrastructure" \

--voice en-US-GuyNeural \
         --write-media output.mp3

List available voices (400+)

edge-tts --list-voices

Popular voices:

Voice ID Language Style

Edge TTS is perfect for content creation, podcast generation, and accessibility — and it's free with no API key required.

🚀

The Latency Challenge

Real-time voice conversation requires end-to-end latency under 500ms (ideally under 300ms) for natural feel:

User speaks → STT (100ms) → LLM thinking (200-500ms) → TTS (100ms) → Audio plays

Total: 400-700ms

GPT-4o approach:

User speaks → Audio tokens in → LLM generates audio tokens out → Decode (50ms)

Total: 250-400ms

The native audio token approach wins because it eliminates two pipeline stages. But it requires the LLM to understand and generate audio directly — a much harder training problem.

🔄

Practical Takeaways

Whisper is the gold standard for STT — open-source, 99 languages, handles noisy audio

Modern TTS uses neural networks end-to-end — from text to phonemes to spectrograms to waveforms

Voice cloning needs just 30 seconds of audio — speaker embeddings capture vocal identity in a vector

GPT-4o generates audio tokens directly — no text-to-speech pipeline, enabling natural conversation

Neural audio codecs (EnCodec) are the foundation — they tokenize audio so LLMs can predict it

Edge TTS is free and production-quality — 400+ neural voices, no API key needed

🏗️

What's Next?

Episode 60: Diffusion Models — How does AI generate images from noise? What are denoising steps, U-Net, and classifier-free guidance? We'll break down how Stable Diffusion and DALL-E turn random static into photorealistic images.

← Previous

Ep 58: Multi-modal AI

Next →

Ep 60: Diffusion Models

Next: Episode 60 — Diffusion Models

Take a photo. Add static noise until nothing remains but TV static. Now reverse the process. That's diffusion.

This is part of a 98-episode series covering AI engineering from tokens to production deployment.