### The pipeline ```mermaid flowchart LR TXT["Text prompt"] --> TE["Text encoder
(CLIP-L or T5)"] TE --> CT["Text
embedding"] NOISE["Noise
4x64x64"] --> UNET["UNet
(denoiser with
cross-attention
to text)"] CT --> UNET UNET --> SCHED["Scheduler
(DPM-Solver++,
Euler)"] SCHED --> LATENT["Clean latent
4x64x64"] LATENT --> VAE["VAE decoder"] VAE --> IMG["512x512
RGB image"] style TE fill:#dbeafe,stroke:#2563eb style UNET fill:#fef3c7,stroke:#d97706 style SCHED fill:#fecaca,stroke:#dc2626 style IMG fill:#dcfce7,stroke:#16a34a ``` - **VAE** — frozen autoencoder. Encoder turns image into latents (used for img2img and training). Decoder turns latents back into an image. - **Text encoder** — CLIP text encoder (SD 1.x/2.x), CLIP-L + CLIP-G (SDXL), or T5-XXL (SD3/FLUX). Produces a sequence of token embeddings. - **U-Net** — the denoiser. Has cross-attention layers that attend from latents to the text embedding at every resolution level. - **Scheduler** — the sampling algorithm (DDIM, Euler, DPM-Solver++). Picks sigmas, blends predicted noise back into the latent. - **Safety checker** — optional NSFW / illegal-content filter on the output image. ### Classifier-free guidance (CFG) Plain text conditioning learns `epsilon_theta(x_t, t, c)` for every prompt `c`. CFG trains the same network with `c` dropped 10% of the time (replaced by an empty embedding), giving a single model that predicts both the conditional and the unconditional noise. At inference: ``` eps = eps_uncond + w * (eps_cond - eps_uncond) ``` `w` is the guidance scale. `w=0` is unconditional, `w=1` is plain conditional, `w>1` pushes the output toward being "more conditioned on the prompt" at the cost of diversity. SD default is `w=7.5`. CFG is the reason text-to-image works at production quality. Without it, prompts bias the output weakly; with it, prompts dominate. ### Latent space geometry The VAE's 4-channel latent is not just a compressed image. It is a manifold where arithmetic roughly corresponds to semantic edits (prompt engineering + interpolation both live here), and where the diffusion U-Net has been trained to spend its entire modelling budget. Decoding a random 4x64x64 latent does not produce a random-looking image — it produces garbage, because only a specific submanifold of latents decodes to valid images. Two consequences: 1. **Img2img** = encode image to latent, add partial noise, run the denoiser, decode. Image structure survives because encoding is near-invertible; content changes based on the prompt. 2. **Inpainting** = same as img2img but the denoiser only updates masked regions; unmasked regions are kept at the encoded latent. ### The U-Net architecture The SD U-Net is a big version of the TinyUNet from Lesson 10 with three additions: - **Transformer blocks** at every spatial resolution, containing self-attention + cross-attention to the text embedding. - **Time embedding** via MLP on sinusoidal encoding. - **Skip connections** between encoder and decoder at matching resolutions. Total parameters in SD 1.5: ~860M. SDXL: ~2.6B. FLUX: ~12B. The jump in params is mostly in attention layers. ### LoRA fine-tuning Full fine-tuning of Stable Diffusion needs 20+ GB of VRAM and updates 860M parameters. LoRA (Low-Rank Adaptation) keeps the base model frozen and injects small rank-decomposition matrices into the attention layers. A LoRA adapter for SD is typically 10-50 MB, trains in 10-60 minutes on a single consumer GPU, and loads at inference time as a drop-in modification. ``` Original: W_q : (d_in, d_out) frozen LoRA: W_q + alpha * (A @ B) where A : (d_in, r), B : (r, d_out) r is typically 4-32. ``` LoRA is how almost every community fine-tune is distributed. CivitAI and Hugging Face host millions of them. ### Schedulers you will see - **DDIM** — deterministic, ~50 steps, simple. - **Euler ancestral** — stochastic, 30-50 steps, slightly more creative samples. - **DPM-Solver++ 2M Karras** — deterministic, 20-30 steps, production default. - **LCM / TCD / Turbo** — consistency models and distilled variants; 1-4 steps at the cost of some quality. Swapping schedulers is a one-line change in `diffusers` and sometimes fixes sample issues without any retraining.