### Three families ```mermaid flowchart LR A["Contrastive
SimCLR, MoCo, CLIP"] --> AT["positive pairs
(same image, 2 augs)
pulled together,
negatives pushed apart"] B["Teacher-student
DINO, BYOL, iBOT"] --> BT["student predicts
teacher's output;
teacher is EMA of student"] C["Masked reconstruction
MAE, BEiT, SimMIM"] --> CT["mask 75% of patches;
reconstruct pixel or
token targets"] style A fill:#dbeafe,stroke:#2563eb style B fill:#fef3c7,stroke:#d97706 style C fill:#dcfce7,stroke:#16a34a ``` ### Contrastive learning (SimCLR) Take one image, apply two random augmentations, get two views. Feed both through the same encoder plus a projection head. Minimise a loss that says "these two embeddings should be close" and "this embedding should be far from every other image's embeddings in the batch." ``` Loss for positive pair (z_i, z_j) among 2N views per batch: L_ij = -log( exp(sim(z_i, z_j) / tau) / sum_k in batch \ {i} exp(sim(z_i, z_k) / tau) ) sim = cosine similarity tau = temperature (0.1 standard) ``` This is the InfoNCE loss. It requires many negatives per positive, so batch size matters — SimCLR needs 512-8192. MoCo introduced a momentum queue of past batches to decouple negative count from batch size. ### Teacher-student (DINO) Two networks with the same architecture: student and teacher. The teacher is an exponential moving average (EMA) of the student's weights. Both see augmented views of the image. The student's output is trained to match the teacher's — no explicit negatives. ``` loss = CE( student_output(view_1), teacher_output(view_2) ) + CE( student_output(view_2), teacher_output(view_1) ) teacher_weights = m * teacher_weights + (1 - m) * student_weights (m ≈ 0.996) ``` Why it does not collapse to "predict a constant": the teacher's output is centred (subtract per-dimension mean) and sharpened (divide by small temperature). Centering prevents one dimension from dominating; sharpening prevents output collapse to uniform. DINO is what DINOv2 scales up, on 142M curated images. The resulting features are the current SOTA for zero-shot visual retrieval and dense prediction. ### Masked reconstruction (MAE) Mask 75% of patches of a ViT input. Pass only the visible 25% through the encoder. A small decoder receives the encoder's output plus mask tokens at masked positions, and is trained to reconstruct the pixels of the masked patches. ``` Encoder: visible 25% of patches -> features Decoder: features + mask tokens at masked positions -> reconstructed pixels Loss: MSE between reconstructed and original pixels on masked patches only ``` Key design choices that make MAE work: - **75% mask ratio** — high. Forces the encoder to learn semantic features; reconstructing 25% would be near-trivial (neighbouring pixels are so correlated that a CNN could nail it). - **Asymmetric encoder/decoder** — the big ViT encoder only sees visible patches; a small decoder (8-layer, 512-dim) handles reconstruction. 3x faster pretraining than naive BEiT. - **Pixel-space reconstruction target** — simpler than BEiT's tokenised target and works better on ViT. After pretraining, discard the decoder. The encoder is the feature extractor. ### Why 75% and not 15% BERT masks 15% of tokens. MAE masks 75%. The difference is information density. - Natural language has high entropy per token. Predicting 15% of tokens is still hard because each masked position has many plausible completions. - Image patches have low entropy — an unmasked neighbourhood often determines the masked patch's pixels almost exactly. To make prediction require semantic understanding, you have to mask aggressively. 75% is high enough that simple spatial extrapolation cannot solve the task; the encoder must represent the image content. ### Linear-probe evaluation After self-supervised pretraining, the standard evaluation is a **linear probe**: freeze the encoder, train a single linear classifier on top on ImageNet labels. Reports top-1 accuracy. - SimCLR ResNet-50: ~71% (2020) - DINO ViT-S/16: ~77% (2021) - MAE ViT-L/16: ~76% (2022) - DINOv2 ViT-g/14: ~86% (2023) Linear probe is a pure measure of feature quality; fine-tuning typically adds 2-5 points but also mixes in the effect of head retraining.