### What a tensor is A tensor is a multi-dimensional array of numbers with a uniform data type. The number of dimensions is the **rank** (or **order**). Each dimension is an **axis**. The **shape** is a tuple listing the size along each axis. ```mermaid graph LR S["Scalar
rank 0
shape: ()"] --> V["Vector
rank 1
shape: (3,)"] V --> M["Matrix
rank 2
shape: (2,3)"] M --> T3["3D Tensor
rank 3
shape: (2,2,2)"] T3 --> T4["4D Tensor
rank 4
shape: (B,C,H,W)"] ``` Total elements = product of all sizes. A shape `(2, 3, 4)` holds `2 * 3 * 4 = 24` elements. ### Tensor shapes in deep learning Different data types map to specific tensor shapes by convention. ```mermaid graph TD subgraph Vision V1["(B, C, H, W)
32, 3, 224, 224"] end subgraph NLP N1["(B, T, D)
16, 128, 768"] end subgraph Attention A1["(B, H, T, D)
16, 12, 128, 64"] end subgraph Weights W1["Linear: (out, in)
Conv2D: (out_c, in_c, kH, kW)
Embedding: (vocab, dim)"] end ``` PyTorch uses NCHW (channels-first). TensorFlow defaults to NHWC (channels-last). Mismatched layouts cause silent slowdowns or errors. ### How memory layout works A 2D array in memory is a 1D sequence of bytes. **Strides** tell you how many elements to skip to move one step along each axis. ```mermaid graph LR subgraph "Row-major (C order)" R["a b c d e f
strides: (3, 1)"] end subgraph "Column-major (F order)" C["a d b e c f
strides: (1, 2)"] end ``` Transpose does not move data. It swaps the strides, making the tensor **non-contiguous** -- the elements for a row are no longer adjacent in memory. ### Broadcasting rules Broadcasting lets you operate on tensors of different shapes without copying data. Align shapes from the right. Two dimensions are compatible when they are equal or one is 1. Fewer dimensions get padded with 1s on the left. ``` Tensor A: (8, 1, 6, 1) Tensor B: (7, 1, 5) Padded B: (1, 7, 1, 5) Result: (8, 7, 6, 5) ``` ### Einsum: the universal tensor operation Einstein summation labels each axis with a letter. Axes in the input but not the output get summed. Axes in both are kept. ```mermaid graph LR subgraph "matmul: ik,kj -> ij" A["A(I,K)"] --> |"sum over k"| C["C(I,J)"] B["B(K,J)"] --> |"sum over k"| C end ``` Key patterns: `i,i->` (dot product), `i,j->ij` (outer product), `ii->` (trace), `ij->ji` (transpose), `bij,bjk->bik` (batch matmul), `bhtd,bhsd->bhts` (attention scores). ```figure tensor-broadcast ```