Skip to main content

Learn Skills Tools Games Exams Newsletter Community

Learn Skills Tools Games Exams Newsletter Community Search

AI From Scratch/Phase 07/Lesson 09/~45 minutes

Vision Transformers (ViT)

BuildPython

An image is a grid of patches. A sentence is a grid of tokens. The same transformer eats both.

Back to phaseTransformers Deep Dive Next lessonAudio Transformers — Whisper Architecture

Phase 07

Transformers Deep Dive

01Why Transformers — The Problems with RNNs 02Self-Attention from Scratch 03Multi-Head Attention 04Positional Encoding — Sinusoidal, RoPE, ALiBi 05The Full Transformer — Encoder + Decoder 06BERT — Masked Language Modeling 07GPT — Causal Language Modeling 08T5, BART — Encoder-Decoder Models 09Vision Transformers (ViT)10Audio Transformers — Whisper Architecture 11Mixture of Experts (MoE)12KV Cache, Flash Attention & Inference Optimization 13Scaling Laws 14Build a Transformer from Scratch — The Capstone 15Attention Variants — Sliding Window, Sparse, Differential 16Speculative Decoding — Draft, Verify, Repeat

Loading lesson page...

Progress

0 / 16 phase lessons0%

Continue target

Phase 07

Transformers Deep Dive

01Why Transformers — The Problems with RNNs 02Self-Attention from Scratch 03Multi-Head Attention 04Positional Encoding — Sinusoidal, RoPE, ALiBi 05The Full Transformer — Encoder + Decoder 06BERT — Masked Language Modeling 07GPT — Causal Language Modeling 08T5, BART — Encoder-Decoder Models 09Vision Transformers (ViT)10Audio Transformers — Whisper Architecture 11Mixture of Experts (MoE)12KV Cache, Flash Attention & Inference Optimization 13Scaling Laws 14Build a Transformer from Scratch — The Capstone 15Attention Variants — Sliding Window, Sparse, Differential 16Speculative Decoding — Draft, Verify, Repeat

On this page

Resources

Dosovitskiy et al. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleReading Touvron et al. (2021). Training data-efficient image transformers & distillation through attentionReading Liu et al. (2021). Swin Transformer: Hierarchical Vision Transformer using Shifted WindowsReading Oquab et al. (2023). DINOv2: Learning Robust Visual Features without SupervisionReading Darcet et al. (2023). Vision Transformers Need RegistersReading Vit ConfiguratorSkill