![Whisper pipeline: audio → mel → encoder → decoder → text](../assets/whisper.svg) ### Step 1 — resample + window Audio at 16 kHz. Clip/pad to 30 seconds. Compute log-mel spectrogram: 80 mel bins, 10 ms stride → ~3,000 frames × 80 features. This is the "input image" that Whisper sees. ### Step 2 — convolutional stem Two Conv1D layers with kernel 3 and stride 2 reduce the 3,000 frames to 1,500. Halves sequence length without adding a lot of parameters. ### Step 3 — encoder A 24-layer (for large) transformer encoder over 1,500 timesteps. Sinusoidal positional encoding, self-attention, GELU FFN. Produces 1,500 × 1,280 hidden states. ### Step 4 — decoder A 24-layer transformer decoder. It autoregressively produces tokens from a BPE vocabulary that is a superset of GPT-2's with a few audio-specific special tokens. ### Step 5 — task tokens The decoder prompt starts with control tokens that tell the model what to do: ``` <|startoftranscript|> <|en|> <|transcribe|> <|0.00|> ``` or ``` <|startoftranscript|> <|fr|> <|translate|> <|0.00|> ``` The model was trained on this convention. You control task by prefix. The 2026 equivalent of instruction-tuning, but applied to speech. ### Step 6 — output Beam search (width 5) with a log-prob threshold. Timestamps are predicted every 0.02 seconds of audio when the `<|notimestamps|>` token is absent. ### Whisper sizes | Model | Params | Layers | d_model | Heads | VRAM (fp16) | |-------|--------|--------|---------|-------|-------------| | Tiny | 39M | 4 | 384 | 6 | ~1 GB | | Base | 74M | 6 | 512 | 8 | ~1 GB | | Small | 244M | 12 | 768 | 12 | ~2 GB | | Medium | 769M | 24 | 1024 | 16 | ~5 GB | | Large | 1550M | 32 | 1280 | 20 | ~10 GB | | Large-v3 | 1550M | 32 | 1280 | 20 | ~10 GB | | Large-v3-turbo | 809M | 32 | 1280 | 20 | ~6 GB (4-layer decoder) | Large-v3-turbo (2024) cut the decoder from 32 layers to 4. 8× faster decoding with <1 WER point regression. That decode speed unlock is why Whisper-turbo is the default for real-time voice agents in 2026. ### What Whisper does not do - No diarization (who is speaking). Pair with pyannote for that. - No real-time streaming natively — the 30-second window is fixed. Modern wrappers (`faster-whisper`, `WhisperX`) bolt on streaming via VAD + overlap. - No long-form context beyond 30 s without external chunking. Works well in practice because human speech rarely needs long-range context for transcription. ### 2026 landscape | Task | Model | Notes | |------|-------|-------| | English ASR | Whisper-turbo, Moonshine | Moonshine is 4× faster on edge | | Multilingual ASR | Whisper-large-v3 | 97 languages | | Streaming ASR | faster-whisper + VAD | 150 ms latency targets achievable | | TTS | Piper, XTTS-v2, Kokoro | Encoder-decoder pattern, but Whisper-shaped | | Audio + language | AudioLM, SeamlessM4T | Text tokens + audio tokens in one transformer |