Scalars, vectors, matrices, tensors

A scalar is a single number, $a\in \mathbb{R}$ — a temperature or a learning rate. A vector is an ordered list of numbers, $\mathbf{x}\in {\mathbb{R}}^d$ — one example's $d$ features. A matrix is a 2-D grid, $A\in {\mathbb{R}}^{m\times n}$ — a whole dataset, or the weights of one neural-network layer.

A tensor generalises this to any number of dimensions (axes). A colour image is a 3-D tensor (height × width × 3 channels); a batch of 32 images is 4-D (32 × height × width × 3). Deep-learning libraries are built around tensors.

Notation used in this course: bold lowercase $\mathbf{x}$ for vectors (column vectors by default), uppercase $X,W$ for matrices, $x_i$ for the $i$-th entry and $A_{ij}$ for row $i$, column $j$. The transpose $A^{\top }$ swaps rows and columns.