### The full preprocessing pipeline at a glance Every production vision system is the same sequence of reversible transforms. Get one step wrong and the model sees a different input than it was trained on. ```mermaid flowchart LR A["Image file
(JPEG/PNG)"] --> B["Decode
uint8 HWC"] B --> C["Convert
colorspace
(RGB/BGR/YCbCr)"] C --> D["Resize
shorter side"] D --> E["Center crop
model size"] E --> F["Divide by 255
float32 [0,1]"] F --> G["Subtract mean
Divide by std"] G --> H["Transpose
HWC → CHW"] H --> I["Batch
CHW → NCHW"] I --> J["Model"] style A fill:#fef3c7,stroke:#d97706 style J fill:#ddd6fe,stroke:#7c3aed style G fill:#fecaca,stroke:#dc2626 style H fill:#bfdbfe,stroke:#2563eb ``` The two red and blue boxes are where 80% of silent failures live: missing standardization and wrong layout. ### A pixel is a sample, not a square A camera sensor counts photons that land on a grid of tiny detectors. Each detector integrates light for a fraction of a second and emits a voltage proportional to how many photons hit it. The sensor then discretizes that voltage into an integer. One detector becomes one pixel. ``` Continuous scene Sensor grid Digital image (infinite detail) (H x W detectors) (H x W integers) ~~~~~ +--+--+--+--+--+ 210 198 180 155 120 ~ ~ ~ | | | | | | 205 195 178 152 118 ~ light ~ ----> +--+--+--+--+--+ ----> 200 190 175 150 115 ~~~~~ | | | | | | 195 185 170 148 112 +--+--+--+--+--+ 188 180 165 145 108 ``` Two choices happen at this step and they fix the ceiling on everything downstream: - **Spatial sampling** decides how many detectors per degree of the scene. Too few, and edges become jagged (aliasing). Too many, and storage and compute explode. - **Intensity quantization** decides how finely the voltage is bucketed. 8 bits gives 256 levels and is standard for display. 10, 12, 16 bits give smoother gradients and matter for medical imaging, HDR, and raw sensor pipelines. A pixel is not a coloured square with area. It is a single measurement. When you resize or rotate, you are resampling that measurement grid. ### Why three channels One detector counts photons across the whole visible spectrum — that is grayscale. To get colour, the sensor covers the grid with a mosaic of red, green, and blue filters. After demosaicing, every spatial location has three integers: the response of the red-filtered detector, green-filtered, and blue-filtered nearby. Those three integers are a pixel's RGB triplet. ``` One pixel in memory: (R, G, B) = (210, 140, 30) <- reddish-orange An H x W RGB image: shape (H, W, 3) stored as H rows of W pixels of 3 values each in [0, 255] for uint8 ``` Three is not magic. Depth cameras add a Z channel. Satellites add infrared and ultraviolet bands. Medical scans often have one channel (X-ray, CT) or many (hyperspectral). The number of channels is the last axis; conv layers learn to mix across it. ### Two layout conventions: HWC and CHW Same tensor, two orderings. Every library picks one. ``` HWC (height, width, channels) CHW (channels, height, width) W -> H -> +-----+-----+-----+ +-----+-----+ H |R G B|R G B|R G B| C |R R R R R R| | +-----+-----+-----+ | +-----+-----+ v |R G B|R G B|R G B| v |G G G G G G| +-----+-----+-----+ +-----+-----+ |B B B B B B| +-----+-----+ PIL, OpenCV, matplotlib, PyTorch, most deep learning almost every image file on disk frameworks, cuDNN kernels ``` CHW exists because convolution kernels slide across H and W. Keeping the channel axis first means each kernel sees a contiguous 2D plane per channel, which vectorizes cleanly. Disk formats keep HWC because that matches how scanlines come out of a sensor. The one-line conversion you will type a thousand times: ``` img_chw = img_hwc.transpose(2, 0, 1) # NumPy img_chw = img_hwc.permute(2, 0, 1) # PyTorch tensor ``` Memory layout, visualised: ```mermaid flowchart TB subgraph HWC["HWC — pixels stored interleaved (PIL, OpenCV, JPEG)"] H1["row 0: R G B | R G B | R G B ..."] H2["row 1: R G B | R G B | R G B ..."] H3["row 2: R G B | R G B | R G B ..."] end subgraph CHW["CHW — channels stored as stacked planes (PyTorch, cuDNN)"] C1["plane R: entire H x W of red values"] C2["plane G: entire H x W of green values"] C3["plane B: entire H x W of blue values"] end HWC -->|"transpose(2, 0, 1)"| CHW CHW -->|"transpose(1, 2, 0)"| HWC ``` ### Byte ranges and dtype Three conventions dominate: | Convention | dtype | Range | Where you see it | |------------|-------|-------|------------------| | Raw | `uint8` | [0, 255] | Files on disk, PIL, OpenCV output | | Normalized | `float32` | [0.0, 1.0] | After `img.astype('float32') / 255` | | Standardized | `float32` | roughly [-2, +2] | After subtracting mean and dividing by std | Convolutional networks were trained on standardized inputs. ImageNet stats `mean=[0.485, 0.456, 0.406]`, `std=[0.229, 0.224, 0.225]` are the arithmetic mean and standard deviation of the three channels over the full ImageNet training set, computed on [0, 1] normalized pixels. Feeding raw `uint8` into a model that expects standardized float is the single most common silent failure in applied vision. ### Color spaces and why they exist RGB is the capture format but it is not always the most useful representation for a model. ``` RGB HSV YCbCr / YUV R red H hue (angle 0-360) Y luminance (brightness) G green S saturation (0-1) Cb chroma blue-yellow B blue V value/brightness (0-1) Cr chroma red-green Linear to Separates color from Separates brightness from sensor output brightness. Useful for color. JPEG and most video color thresholding, UI codecs compress the chroma sliders, simple filters channels harder because the human eye is less sensitive to chroma detail than to Y. ``` For most modern CNNs you feed RGB. You meet other spaces when: - **HSV** — classical CV code, color-based segmentation, white-balancing. - **YCbCr** — reading JPEG internals, video pipelines, super-resolution models that operate on Y only. - **Grayscale** — OCR, document models, any case where color is nuisance variable rather than signal. Grayscale from RGB is a weighted sum, not an average, because the human eye is more sensitive to green than to red or blue: ``` Y = 0.299 R + 0.587 G + 0.114 B (ITU-R BT.601, the classic weights) ``` ### Aspect ratio, resizing, and interpolation Every model has a fixed input size (224x224 for most ImageNet classifiers, 384x384 or 512x512 for modern detectors). Your images rarely match. The three resize choices that matter: - **Resize shorter side, then center crop** — the standard ImageNet recipe. Preserves aspect ratio, throws away a strip of edge pixels. - **Resize and pad** — preserves aspect ratio and every pixel, adds black bars. Standard for detection and OCR. - **Resize directly to target** — stretches the image. Cheap, distorts geometry, fine for many classification tasks. The interpolation method decides how intermediate pixels are computed when the new grid does not align with the old one: ``` Nearest neighbour fastest, blocky, only choice for masks/labels Bilinear fast, smooth, default for most image resizing Bicubic slower, sharper on upscaling Lanczos slowest, best quality, used for final display ``` Rule of thumb: bilinear for training, bicubic or lanczos for assets you will look at, nearest for anything containing integer class IDs. ```figure conv-output-size ```