The covariance matrix

For centred data $X$ ($n\times d$), the covariance matrix is $\Sigma =\frac{1}{n}{X}^{\top }X$, a $d\times d$ matrix. Entry ${\Sigma }_{jj}$ is the variance of feature $j$; entry ${\Sigma }_{jk}$ is the covariance between features $j$ and $k$.

It is symmetric (${\Sigma }_{jk}={\Sigma }_{kj}$) and positive semi-definite (${\mathbf{v}}^{\top }\Sigma \mathbf{v}\ge 0$ for any $\mathbf{v}$). Those two facts guarantee real, non-negative eigenvalues and orthogonal eigenvectors — exactly what PCA needs.

The variance of the data projected onto a unit direction $\mathbf{v}$ is ${\mathbf{v}}^{\top }\Sigma \mathbf{v}$. That single expression connects the covariance matrix to 'how spread out the data is along $\mathbf{v}$'.