Principal Component Analysis

Principal component analysis (PCA) finds a new set of orthogonal axes, ordered by how much of the data's variance lies along them. Keeping only the first $k$ axes gives the best $k$-dimensional linear summary of the data.

The recipe:

• Centre the data (subtract the mean of each feature), and usually standardise it.\n- Compute the covariance matrix $\Sigma =\frac{1}{n}{X}^{\top }X$.\n- Find its eigenvectors (principal directions) and eigenvalues (variance along each).\n- Sort by eigenvalue and keep the top $k$ eigenvectors as columns of $W_k$.\n- Project: $Z=X{W}_k$, an $n\times k$ matrix of new features.

The components are uncorrelated with each other. PCA is unsupervised (it ignores labels), linear, and sensitive to scaling — a feature measured in large units would dominate unless standardised.