### The Module Abstraction Every layer in PyTorch inherits from `nn.Module`. A Module has three responsibilities: 1. **forward()** -- compute the output given inputs 2. **parameters()** -- return all trainable weights 3. **backward()** -- compute gradients (handled by autograd in PyTorch, explicit in ours) A Linear layer is a Module. A ReLU activation is a Module. A dropout layer is a Module. A batch normalization layer is a Module. They all have the same interface. ### Sequential Container `nn.Sequential` chains Modules. Forward pass: feed data through Module 1, then Module 2, then Module 3. Backward pass: reverse the chain. The container itself is a Module -- it has forward(), parameters(), and backward(). This is the composite pattern: a sequence of Modules is itself a Module. ### Training vs Evaluation Mode Dropout randomly zeroes neurons during training but passes everything through during evaluation. Batch normalization uses batch statistics during training but running averages during evaluation. The `train()` and `eval()` methods toggle this behavior. Every Module has a `training` flag. ### Optimizer The optimizer updates parameters using their gradients. SGD: `param -= lr * grad`. Adam: maintains momentum and variance estimates, then updates. The optimizer does not know about the network architecture -- it only sees a flat list of parameters and their gradients. ### DataLoader Batching matters for two reasons. First, you cannot fit the entire dataset in memory for large problems. Second, mini-batch gradient descent provides noise that helps escape local minima. The DataLoader splits data into batches and optionally shuffles between epochs. ### Framework Architecture ```mermaid graph TD subgraph "Modules" Linear["Linear
W*x + b"] ReLU["ReLU
max(0, x)"] Sigmoid["Sigmoid
1/(1+e^-x)"] Dropout["Dropout
random zero mask"] BatchNorm["BatchNorm
normalize activations"] end subgraph "Containers" Sequential["Sequential
chains modules"] end subgraph "Loss Functions" MSE["MSELoss
(pred - target)^2"] BCE["BCELoss
binary cross-entropy"] end subgraph "Optimizers" SGD["SGD
param -= lr * grad"] Adam["Adam
adaptive moments"] end subgraph "Data" DataLoader["DataLoader
batching + shuffle"] end Sequential --> |"contains"| Linear Sequential --> |"contains"| ReLU Sequential --> |"forward/backward"| MSE SGD --> |"updates"| Sequential DataLoader --> |"feeds"| Sequential ``` ### Training Loop ```mermaid sequenceDiagram participant DL as DataLoader participant M as Model participant L as Loss participant O as Optimizer loop Each Epoch DL->>M: batch of inputs M->>M: forward pass (layer by layer) M->>L: predictions L->>L: compute loss L->>M: backward pass (gradients) M->>O: parameters + gradients O->>M: updated parameters O->>O: zero gradients end ``` ### Module Hierarchy ```mermaid classDiagram class Module { +forward(x) +backward(grad) +parameters() +train() +eval() } class Linear { -weights -biases +forward(x) +backward(grad) } class ReLU { +forward(x) +backward(grad) } class Sequential { -modules[] +forward(x) +backward(grad) +parameters() } Module <|-- Linear Module <|-- ReLU Module <|-- Sequential Sequential *-- Module ``` ```figure gradient-clipping ```