### Stochastic Gradient Descent (SGD) The simplest optimizer. Compute the gradient on a mini-batch and step in the opposite direction. ``` w = w - lr * gradient ``` The "stochastic" means you use a random subset (mini-batch) of data to estimate the gradient, rather than the full dataset. This noise is actually useful -- it helps escape sharp local minima. But the noise also causes oscillation. Learning rate is the only knob. Too high: the loss diverges. Too low: training takes forever. The optimal value depends on the architecture, the data, the batch size, and the current stage of training. For vanilla SGD on modern networks, typical values range from 0.01 to 0.1. But even within a single training run, the ideal learning rate changes. ### Momentum The ball-rolling-downhill analogy is overused but accurate. Instead of stepping by the gradient alone, you maintain a velocity that accumulates past gradients. ``` m_t = beta * m_{t-1} + gradient w = w - lr * m_t ``` Beta (typically 0.9) controls how much history to keep. With beta = 0.9, the momentum is roughly the average of the last 10 gradients (1 / (1 - 0.9) = 10). Why this fixes oscillation: gradients that point in the same direction accumulate. Gradients that flip direction cancel out. In that narrow valley, the "across" component flips sign each step and gets dampened. The "along" component stays consistent and gets amplified. The result is smooth acceleration in the useful direction. Real numbers: SGD alone on a badly conditioned loss landscape might take 10,000 steps. SGD with momentum (beta=0.9) typically takes 3,000-5,000 steps on the same problem. The speedup is not marginal. ### RMSProp The first per-parameter adaptive learning rate method that actually worked. Proposed by Hinton in a Coursera lecture (never formally published). ``` s_t = beta * s_{t-1} + (1 - beta) * gradient^2 w = w - lr * gradient / (sqrt(s_t) + epsilon) ``` s_t tracks the running average of squared gradients. Parameters with consistently large gradients get divided by a large number (smaller effective learning rate). Parameters with small gradients get divided by a small number (larger effective learning rate). This solves the "one learning rate for all parameters" problem. A weight that's already been getting large updates is probably near its target -- slow it down. A weight that's been getting tiny updates might be undertrained -- speed it up. Epsilon (typically 1e-8) prevents division by zero when a parameter hasn't been updated. ### Adam: Momentum + RMSProp Adam combines both ideas. It maintains two exponential moving averages per parameter: ``` m_t = beta1 * m_{t-1} + (1 - beta1) * gradient (first moment: mean) v_t = beta2 * v_{t-1} + (1 - beta2) * gradient^2 (second moment: variance) ``` **Bias correction** is the key detail most explanations skip. At step 1, m_1 = (1 - beta1) * gradient. With beta1 = 0.9, that's 0.1 * gradient -- ten times too small. The moving average hasn't warmed up yet. Bias correction compensates: ``` m_hat = m_t / (1 - beta1^t) v_hat = v_t / (1 - beta2^t) ``` At step 1 with beta1 = 0.9: m_hat = m_1 / (1 - 0.9) = m_1 / 0.1 = the actual gradient. At step 100: (1 - 0.9^100) is approximately 1.0, so the correction vanishes. Bias correction matters for the first ~10 steps and is irrelevant after ~50. The update: ``` w = w - lr * m_hat / (sqrt(v_hat) + epsilon) ``` Adam defaults: lr = 0.001, beta1 = 0.9, beta2 = 0.999, epsilon = 1e-8. These defaults work for 80% of problems. When they don't, change lr first. Then beta2. Almost never change beta1 or epsilon. ### AdamW: Weight Decay Done Right L2 regularization adds lambda * w^2 to the loss. In vanilla SGD, this is equivalent to weight decay (subtracting lambda * w from the weight at each step). In Adam, this equivalence breaks. The Loshchilov & Hutter insight: when you add L2 to the loss and then Adam processes the gradient, the adaptive learning rate scales the regularization term too. Parameters with large gradient variance get less regularization. Parameters with small variance get more. This is not what you want -- you want uniform regularization regardless of the gradient statistics. AdamW fixes this by applying weight decay directly to the weights, after the Adam update: ``` w = w - lr * m_hat / (sqrt(v_hat) + epsilon) - lr * lambda * w ``` The weight decay term (lr * lambda * w) is not scaled by Adam's adaptive factor. Every parameter gets the same proportional shrinkage. This seems like a minor detail. It's not. AdamW converges to better solutions than Adam + L2 regularization on virtually every task. It's the default optimizer in PyTorch for training transformers, diffusion models, and most modern architectures. BERT, GPT, LLaMA, Stable Diffusion -- all trained with AdamW. ### Learning Rate: The Most Important Hyperparameter ```mermaid graph TD LR["Learning Rate"] --> TooHigh["Too high (lr > 0.01)"] LR --> JustRight["Just right"] LR --> TooLow["Too low (lr < 0.00001)"] TooHigh --> Diverge["Loss explodes
NaN weights
Training crashes"] JustRight --> Converge["Loss decreases steadily
Reaches good minimum
Generalizes well"] TooLow --> Stall["Loss decreases slowly
Gets stuck in suboptimal minimum
Wastes compute"] JustRight --> Schedule["Usually needs scheduling"] Schedule --> Warmup["Warmup: ramp from 0 to max
First 1-10% of training"] Schedule --> Decay["Decay: reduce over time
Cosine or linear"] ``` If you tune one hyperparameter, tune the learning rate. A 10x change in learning rate matters more than any architectural decision you'll make. Common defaults: - SGD: lr = 0.01 to 0.1 - Adam/AdamW: lr = 1e-4 to 3e-4 - Fine-tuning pretrained models: lr = 1e-5 to 5e-5 - Learning rate warmup: linear ramp over first 1-10% of steps ### Optimizer Comparison ```mermaid flowchart LR subgraph "Optimization Path" SGD_P["SGD
Oscillates across valley
Slow but finds flat minima"] Mom_P["SGD + Momentum
Smoother path
3x faster than SGD"] Adam_P["Adam
Adapts per-parameter
Fast convergence"] AdamW_P["AdamW
Adam + proper decay
Best generalization"] end SGD_P --> Mom_P --> Adam_P --> AdamW_P ``` ### When Each Optimizer Wins ```mermaid flowchart TD Task["What are you training?"] --> Type{"Model type?"} Type -->|"Transformer / LLM"| AdamW["AdamW
lr=1e-4, wd=0.01-0.1"] Type -->|"CNN / ResNet"| SGD_M["SGD + Momentum
lr=0.1, momentum=0.9"] Type -->|"GAN"| Adam2["Adam
lr=2e-4, beta1=0.5"] Type -->|"Fine-tuning"| AdamW2["AdamW
lr=2e-5, wd=0.01"] Type -->|"Don't know yet"| Default["Start with AdamW
lr=3e-4, wd=0.01"] ``` ```figure optimizer-trajectory ```