Matrix multiplication

The product $C=AB$ of $A\in {\mathbb{R}}^{m\times n}$ and $B\in {\mathbb{R}}^{n\times p}$ is $C\in {\mathbb{R}}^{m\times p}$ with $C_{ij}={\sum }_k{A}_{ik}{B}_{kj}$: entry $(i,j)$ is the dot product of row $i$ of $A$ with column $j$ of $B$. The inner dimensions must match ($n$ and $n$).

Matrix multiplication is not commutative — $AB\ne BA$ in general, and $BA$ may not even be defined. It is associative, $(AB)C=A(BC)$, and $(AB{)}^{\top }=B^{\top }{A}^{\top }$.

A matrix is a linear map: $A\mathbf{x}$ transforms vector $\mathbf{x}$. Multiplying matrices composes maps — first apply $B$, then $A$. This is why a neural network without activation functions collapses to a single matrix: $W_2(W_1\mathbf{x})=(W_2{W}_1)\mathbf{x}$.

Predicting for a whole dataset at once is one matrix product: $\hat{\mathbf{y}}=X\mathbf{w}$, with $X$ of shape $n\times d$ and $\mathbf{w}$ of shape $d\times 1$. This vectorisation is why GPUs make ML fast.