Covariance

Covariance measures how two variables move together: $\mathrm{Cov}(X,Y)=\mathbb{E}[(X-{\mu }_X)(Y-{\mu }_Y)]$. Positive: when one is above its mean, the other tends to be too. Negative: they move in opposite directions. Zero: no *linear* relationship.

Because covariance depends on units, we often normalise it into the correlation $\rho =\mathrm{Cov}(X,Y)/({\sigma }_X{\sigma }_Y)$, which always lies in $[-1,1]$.

For a vector of $d$ features, the covariance matrix $\Sigma$ collects every pair: ${\Sigma }_{ij}=\mathrm{Cov}(X_i,X_j)$, with variances on the diagonal. It is symmetric and positive semi-definite, and it is the matrix PCA diagonalises.