### Semantic vs instance vs panoptic ```mermaid flowchart LR IN["Input image"] --> SEM["Semantic
(pixel → class)"] IN --> INS["Instance
(pixel → object id,
only foreground classes)"] IN --> PAN["Panoptic
(every pixel → class + id)"] style SEM fill:#dbeafe,stroke:#2563eb style INS fill:#fef3c7,stroke:#d97706 style PAN fill:#dcfce7,stroke:#16a34a ``` - **Semantic** says "this pixel is road, that pixel is car." Two cars next to each other collapse into a single blob. - **Instance** says "this pixel is car #3, that pixel is car #5." Ignores background stuff ("stuff" = sky, road, grass). - **Panoptic** unifies both: every pixel gets a class label, every instance gets a unique id, stuff and things both segmented. This lesson covers semantic. The next lesson (Mask R-CNN) covers instance. ### The U-Net shape ```mermaid flowchart LR subgraph ENC["Encoder (contracting)"] E1["64
H x W"] --> E2["128
H/2 x W/2"] E2 --> E3["256
H/4 x W/4"] E3 --> E4["512
H/8 x W/8"] end subgraph BOT["Bottleneck"] B1["1024
H/16 x W/16"] end subgraph DEC["Decoder (expanding)"] D4["512
H/8 x W/8"] --> D3["256
H/4 x W/4"] D3 --> D2["128
H/2 x W/2"] D2 --> D1["64
H x W"] end E4 --> B1 --> D4 E1 -. skip .-> D1 E2 -. skip .-> D2 E3 -. skip .-> D3 E4 -. skip .-> D4 D1 --> OUT["1x1 conv
classes"] style ENC fill:#dbeafe,stroke:#2563eb style BOT fill:#fef3c7,stroke:#d97706 style DEC fill:#dcfce7,stroke:#16a34a ``` The encoder halves spatial resolution four times and doubles channels. The decoder reverses: doubles spatial resolution four times and halves channels. The skip connections concatenate matching encoder features with decoder features at every resolution. The final 1x1 conv maps `64 -> num_classes` at full resolution. Why skip connections are necessary: the decoder has seen only small feature maps by the time it tries to output pixel-level predictions. Without the skips it cannot localise edges accurately because that information was compressed away in the encoder. Skip connections hand it the high-resolution feature maps the encoder computed on the way down. ### Transposed vs bilinear upsample The decoder has to expand spatial dimensions. Two options: - **Transposed convolution** (`nn.ConvTranspose2d`) — learnable upsample. Historical U-Net default. Can produce checkerboard artifacts if stride and kernel size do not divide evenly. - **Bilinear upsample + 3x3 conv** — smooth upsample followed by a conv. Fewer artifacts, fewer parameters, now the modern default. Both appear in the wild. For a first U-Net, bilinear is safer. ### Cross-entropy on a pixel grid For semantic segmentation with C classes, the model output is `(N, C, H, W)`. The target is `(N, H, W)` with integer class IDs. Cross-entropy is identical to the classification case, just applied at every spatial position: ``` Loss = mean over (n, h, w) of -log( softmax(logits[n, :, h, w])[target[n, h, w]] ) ``` `F.cross_entropy` in PyTorch handles this shape natively. No reshape needed. ### Dice loss and why you need it Cross-entropy treats every pixel equally. That is wrong when one class dominates the frame (medical imaging: 99% background, 1% tumour). The network can score 99% accuracy by predicting background everywhere and still be useless. Dice loss solves this by directly optimising the overlap between predicted and true mask: ``` Dice(p, y) = 2 * sum(p * y) / (sum(p) + sum(y) + epsilon) Dice_loss = 1 - Dice ``` where `p` is the sigmoid/softmax probability map for a class and `y` is the binary ground-truth mask. The loss is zero only when the overlap is perfect. Because it is ratio-based, class imbalance is irrelevant. In practice, use the **combined loss**: ``` L = L_cross_entropy + lambda * L_dice (lambda ~ 1) ``` Cross-entropy gives stable gradients early in training; Dice focuses the tail of training on actually matching the mask shape. This combination is the medical-imaging default and hard to beat on any class-imbalanced dataset. ### Evaluation metrics - **Pixel accuracy** — percent of pixels predicted correctly. Cheap. Broken on imbalanced data for the same reason as accuracy in classification. - **IoU per class** — intersection over union for each class's mask; average across classes = mIoU. - **Dice (F1 on pixels)** — similar to IoU; `Dice = 2 * IoU / (1 + IoU)`. Medical imaging prefers Dice, driving community prefers IoU; they are monotonically related. - **Boundary F1** — measures how close predicted boundaries are to ground-truth boundaries, penalising even small shifts. Important for high-precision tasks like semiconductor inspection. Report IoU per class, not just mIoU. Mean IoU hides a class at 15% when nine others are at 85%. ### Input resolution trade-off U-Net's encoder halves resolution four times, so the input must be divisible by 16. Medical images are often 512x512 or 1024x1024. Autonomous-driving crops are 2048x1024. The memory cost of U-Net scales with `H * W * C_max`, and at 1024x1024 with 1024 bottleneck channels the forward pass already uses gigabytes of VRAM. Two standard workarounds: 1. Tile the input — process 256x256 tiles with overlap and stitch. 2. Replace the bottleneck with dilated convolutions that keep spatial resolution higher but widen receptive field (the DeepLab family). For a first model, a 256x256 input with a 64-channel-base U-Net trains comfortably on 8 GB VRAM.