### Constant Learning Rate The simplest approach. Pick a number, use it for every step. ``` lr(t) = lr_0 ``` Rarely optimal. It's either too high for the end of training (oscillation around the minimum) or too low for the beginning (wasted compute on tiny steps). Works fine for small models and debugging. A terrible choice for anything that trains for more than an hour. ### Step Decay The old-school approach from the ResNet era. Cut the learning rate by a factor (usually 10x) at fixed epochs. ``` lr(t) = lr_0 * gamma^(floor(epoch / step_size)) ``` Where gamma = 0.1 and step_size = 30 means: lr drops by 10x every 30 epochs. ResNet-50 used this -- lr=0.1, drop by 10x at epochs 30, 60, and 90. The problem: the optimal decay points depend on the dataset and architecture. Move to a different problem and you need to re-tune when to drop. The transitions are abrupt -- loss can spike when the rate suddenly changes. ### Cosine Annealing Smooth decay from the maximum learning rate to a minimum, following a cosine curve: ``` lr(t) = lr_min + 0.5 * (lr_max - lr_min) * (1 + cos(pi * t / T)) ``` Where t is the current step and T is the total number of steps. At t=0, the cosine term is 1, so lr = lr_max. At t=T, the cosine term is -1, so lr = lr_min. The decay is gentle at first, accelerates in the middle, and becomes gentle again near the end. This is the default for most modern training runs. No hyperparameters to tune beyond lr_max and lr_min. The cosine shape matches the empirical observation that most learning happens in the middle of training -- you want reasonable step sizes during that critical period. ### Warmup: Why You Start Small Adam and other adaptive optimizers maintain running estimates of gradient mean and variance. At step 0, these estimates are initialized to zero. The first few gradient updates are based on garbage statistics. If your learning rate is large during this period, the model takes huge, poorly-directed steps. Warmup fixes this. Start with a tiny learning rate (often lr_max / warmup_steps or even zero) and linearly ramp up to lr_max over the first N steps. By the time you reach the full learning rate, Adam's statistics have stabilized. ``` lr(t) = lr_max * (t / warmup_steps) for t < warmup_steps ``` Typical warmup: 1-5% of total training steps. Llama 3 trained for ~1.8 trillion tokens and warmed up for 2000 steps. GPT-3 warmed up over 375 million tokens. ### Linear Warmup + Cosine Decay The modern default. Ramp up linearly, then decay with cosine: ``` if t < warmup_steps: lr(t) = lr_max * (t / warmup_steps) else: progress = (t - warmup_steps) / (total_steps - warmup_steps) lr(t) = lr_min + 0.5 * (lr_max - lr_min) * (1 + cos(pi * progress)) ``` This is what Llama, GPT, PaLM, and most modern transformers use. The warmup prevents early instability. The cosine decay settles the model into a good minimum. ### 1cycle Policy Leslie Smith's discovery (2018): ramp the learning rate up from a low value to a high value in the first half of training, then ramp it back down in the second half. Counterintuitive -- why would you *increase* the learning rate midway through? The theory: a high learning rate acts as regularization by adding noise to the optimization trajectory. The model explores more of the loss landscape during the ramp-up phase, finding better basins. The ramp-down phase then refines within the best basin found. ``` Phase 1 (0 to T/2): lr ramps from lr_max/25 to lr_max Phase 2 (T/2 to T): lr ramps from lr_max to lr_max/10000 ``` 1cycle often trains faster than cosine annealing for a fixed compute budget. The tradeoff: you must know the total number of steps in advance. ### Schedule Shapes ```mermaid graph LR subgraph "Constant" C1["lr"] --- C2["lr"] --- C3["lr"] end subgraph "Step Decay" S1["0.1"] --- S2["0.1"] --- S3["0.01"] --- S4["0.001"] end subgraph "Cosine Annealing" CS1["lr_max"] --> CS2["gradual"] --> CS3["steep"] --> CS4["lr_min"] end subgraph "Warmup + Cosine" WC1["0"] --> WC2["lr_max"] --> WC3["cosine"] --> WC4["lr_min"] end ``` ### Decision Flowchart ```mermaid flowchart TD Start["Choosing a LR schedule"] --> Know{"Know total
training steps?"} Know -->|"Yes"| Budget{"Compute budget?"} Know -->|"No"| Constant["Use constant LR
with manual decay"] Budget -->|"Large (days/weeks)"| WarmCos["Warmup + Cosine Decay
(Llama/GPT default)"] Budget -->|"Small (hours)"| OneCycle["1cycle Policy
(fastest convergence)"] Budget -->|"Moderate"| Cosine["Cosine Annealing
(safe default)"] WarmCos --> Warmup["Warmup = 1-5% of steps"] OneCycle --> FindLR["Find lr_max with LR range test"] Cosine --> MinLR["Set lr_min = lr_max / 10"] ``` ### Real Numbers from Published Models ```mermaid graph TD subgraph "Published LR Configs" L3["Llama 3 (405B)
Peak: 3e-4
Warmup: 2000 steps
Schedule: Cosine to 3e-5"] G3["GPT-3 (175B)
Peak: 6e-4
Warmup: 375M tokens
Schedule: Cosine to 0"] R50["ResNet-50
Peak: 0.1
Warmup: none
Schedule: Step decay x0.1 at 30,60,90"] B["BERT (340M)
Peak: 1e-4
Warmup: 10K steps
Schedule: Linear decay"] end ``` ```figure lr-schedule ```