Skip to main content

Learn Tools Games Exams Newsletter Community

Learn Tools Games Exams Newsletter Community Search

Phase 07: Transformers Deep Dive

AI From Scratch/Lesson 14/~120 minutes

Build a Transformer from Scratch — The Capstone

Thirteen lessons. One model. No shortcuts.

BuildPython

Back to phaseTransformers Deep Dive Next lessonAttention Variants — Sliding Window, Sparse, Differential

Phase 07

Transformers Deep Dive

01Why Transformers — The Problems with RNNs 02Self-Attention from Scratch 03Multi-Head Attention 04Positional Encoding — Sinusoidal, RoPE, ALiBi 05The Full Transformer — Encoder + Decoder 06BERT — Masked Language Modeling 07GPT — Causal Language Modeling 08T5, BART — Encoder-Decoder Models 09Vision Transformers (ViT)10Audio Transformers — Whisper Architecture 11Mixture of Experts (MoE)12KV Cache, Flash Attention & Inference Optimization 13Scaling Laws 14Build a Transformer from Scratch — The Capstone 15Attention Variants — Sliding Window, Sparse, Differential 16Speculative Decoding — Draft, Verify, Repeat

Loading lesson page...

Progress

0 / 16 phase lessons0%

Continue target

Phase 07

Transformers Deep Dive

01Why Transformers — The Problems with RNNs 02Self-Attention from Scratch 03Multi-Head Attention 04Positional Encoding — Sinusoidal, RoPE, ALiBi 05The Full Transformer — Encoder + Decoder 06BERT — Masked Language Modeling 07GPT — Causal Language Modeling 08T5, BART — Encoder-Decoder Models 09Vision Transformers (ViT)10Audio Transformers — Whisper Architecture 11Mixture of Experts (MoE)12KV Cache, Flash Attention & Inference Optimization 13Scaling Laws 14Build a Transformer from Scratch — The Capstone 15Attention Variants — Sliding Window, Sparse, Differential 16Speculative Decoding — Draft, Verify, Repeat

On this page

Resources

The Annotated Transformer (Harvard NLP)Reading Transformer ReviewSkill main.pyCode Source lessonSource