### The Overfitting Spectrum Every model sits somewhere on a spectrum from underfitting (too simple to capture the pattern) to overfitting (so complex it captures noise). The sweet spot is in between, and regularization pushes models toward it from the overfit side. ```mermaid graph LR Under["Underfitting
Train: 60%
Test: 58%
Model too simple"] --> Good["Good Fit
Train: 95%
Test: 92%
Generalizes well"] Good --> Over["Overfitting
Train: 99.9%
Test: 65%
Memorized noise"] Dropout["Dropout"] -->|"Pushes left"| Over WD["Weight Decay"] -->|"Pushes left"| Over BN["BatchNorm"] -->|"Pushes left"| Over Aug["Data Augmentation"] -->|"Pushes left"| Over ``` ### Dropout The simplest regularization technique with the most elegant interpretation. During training, randomly set each neuron's output to zero with probability p. ``` output = activation(z) * mask where mask[i] ~ Bernoulli(1 - p) ``` With p = 0.5, half the neurons are zeroed on every forward pass. The network must learn redundant representations because it can't predict which neurons will be available. This prevents co-adaptation -- neurons learning to rely on specific other neurons being present. The ensemble interpretation: a network with N neurons and dropout creates 2^N possible subnetworks (every combination of which neurons are on or off). Training with dropout approximately trains all 2^N subnetworks simultaneously, each on different mini-batches. At test time, you use all neurons (no dropout) and scale outputs by (1 - p) to match the expected value during training. This is equivalent to averaging the predictions of 2^N subnetworks -- a massive ensemble from a single model. In practice, the scaling is applied during training instead of testing (inverted dropout): ``` During training: output = activation(z) * mask / (1 - p) During testing: output = activation(z) (no change needed) ``` This is cleaner because test code doesn't need to know about dropout at all. Default rates: p = 0.1 for transformers, p = 0.5 for MLPs, p = 0.2-0.3 for CNNs. Higher dropout = stronger regularization = more underfitting risk. ### Weight Decay (L2 Regularization) Add the squared magnitude of all weights to the loss: ``` total_loss = task_loss + (lambda / 2) * sum(w_i^2) ``` The gradient of the regularization term is lambda * w. This means at every step, each weight is shrunk toward zero by a fraction proportional to its magnitude. Large weights get penalized more. The model is pushed toward solutions where no single weight dominates. Why this helps generalization: overfit models tend to have large weights that amplify noise in the training data. Weight decay keeps weights small, which limits the model's effective capacity and forces it to rely on robust, generalizable features rather than memorized quirks. The lambda hyperparameter controls the strength. Typical values: - 0.01 for AdamW on transformers - 1e-4 for SGD on CNNs - 0.1 for heavily overfit models As discussed in lesson 06: weight decay and L2 regularization are equivalent in SGD but not in Adam. Always use AdamW (decoupled weight decay) when training with Adam. ### Batch Normalization Normalize the output of each layer across the mini-batch before passing it to the next layer. For a mini-batch of activations at some layer: ``` mu = (1/B) * sum(x_i) (batch mean) sigma^2 = (1/B) * sum((x_i - mu)^2) (batch variance) x_hat = (x_i - mu) / sqrt(sigma^2 + eps) (normalize) y = gamma * x_hat + beta (scale and shift) ``` Gamma and beta are learnable parameters that let the network undo the normalization if that's optimal. Without them, you'd be forcing every layer's output to be zero-mean unit-variance, which might not be what the network wants. **Training vs inference split:** During training, mu and sigma come from the current mini-batch. During inference, you use running averages accumulated during training (exponential moving average with momentum = 0.1, meaning 90% old + 10% new). Why BatchNorm works is still debated. The original paper claimed it reduces "internal covariate shift" (the distribution of layer inputs changing as earlier layers update). Santurkar et al. (2018) showed this explanation is wrong. The actual reason: BatchNorm makes the loss landscape smoother. The gradients are more predictive, the Lipschitz constants are smaller, and the optimizer can take larger steps safely. This is why BatchNorm lets you use higher learning rates and converge faster. BatchNorm has a fundamental limitation: it depends on batch statistics. With batch size 1, the mean and variance are meaningless. With small batches (< 32), the statistics are noisy and hurt performance. This matters for tasks like object detection (where memory limits batch size) and language modeling (where sequence lengths vary). ### Layer Normalization Normalize across features instead of across the batch. For a single sample: ``` mu = (1/D) * sum(x_j) (feature mean) sigma^2 = (1/D) * sum((x_j - mu)^2) (feature variance) x_hat = (x_j - mu) / sqrt(sigma^2 + eps) y = gamma * x_hat + beta ``` D is the feature dimension. Each sample is normalized independently -- no dependence on batch size. This is why transformers use LayerNorm instead of BatchNorm. Sequences have variable lengths, batch sizes are often small (or 1 during generation), and the computation is identical between training and inference. LayerNorm in transformers is applied after each self-attention block and each feed-forward block (Post-LN), or before them (Pre-LN, which is more stable for training). ### RMSNorm LayerNorm without the mean subtraction. Proposed by Zhang & Sennrich (2019). ``` rms = sqrt((1/D) * sum(x_j^2)) y = gamma * x / rms ``` That's it. No mean computation, no beta parameter. The observation: the re-centering (mean subtraction) in LayerNorm contributes very little to the model's performance, but costs computation. Removing it gives the same accuracy with about 10% less overhead. LLaMA, LLaMA 2, LLaMA 3, Mistral, and most modern LLMs use RMSNorm instead of LayerNorm. At the scale of billions of parameters and trillions of tokens, that 10% savings is significant. ### Normalization Comparison ```mermaid graph TD subgraph "Batch Normalization" BN_D["Normalize across BATCH
for each feature"] BN_S["Batch: [x1, x2, x3, x4]
Feature 1: normalize [x1f1, x2f1, x3f1, x4f1]"] BN_P["Needs batch > 32
Different train vs eval
Used in CNNs"] end subgraph "Layer Normalization" LN_D["Normalize across FEATURES
for each sample"] LN_S["Sample x1: normalize [f1, f2, f3, f4]"] LN_P["Batch-independent
Same train vs eval
Used in Transformers"] end subgraph "RMS Normalization" RN_D["Like LayerNorm
but skip mean subtraction"] RN_S["Just divide by RMS
No centering"] RN_P["10% faster than LayerNorm
Same accuracy
Used in LLaMA, Mistral"] end ``` ### Data Augmentation as Regularization Not a model modification but a data modification. Transform training inputs while preserving labels: - Images: random crop, flip, rotation, color jitter, cutout - Text: synonym replacement, back-translation, random deletion - Audio: time stretch, pitch shift, noise addition The effect is identical to regularization: it increases the effective size of the training set, making it harder for the model to memorize specific examples. A model that only sees each image once in its original form can memorize it. A model that sees 50 augmented versions of each image is forced to learn the invariant structure. ### Early Stopping The simplest regularizer: stop training when validation loss starts increasing. The model hasn't overfit yet at that point. In practice, you track validation loss every epoch, save the best model, and continue training for a "patience" window (typically 5-20 epochs). If validation loss doesn't improve within the patience window, you stop and load the best saved model. ### When to Apply What ```mermaid flowchart TD Gap{"Train-test
accuracy gap?"} -->|"> 10%"| Heavy["Heavy regularization"] Gap -->|"5-10%"| Medium["Moderate regularization"] Gap -->|"< 5%"| Light["Light regularization"] Heavy --> D5["Dropout p=0.3-0.5"] Heavy --> WD2["Weight decay 0.01-0.1"] Heavy --> Aug["Aggressive data augmentation"] Heavy --> ES["Early stopping"] Medium --> D3["Dropout p=0.1-0.2"] Medium --> WD1["Weight decay 0.001-0.01"] Medium --> Norm["BatchNorm or LayerNorm"] Light --> D1["Dropout p=0.05-0.1"] Light --> WD0["Weight decay 1e-4"] ``` ```figure l2-regularization ```