PCA via SVD · Machine Learning

PCA via SVD

The numerically stable way to compute components.

In practice, PCA is computed with the singular value decomposition of the centred data matrix: $X = U S V^{⊤}$ , where $U$ ( $n \times n$ ) and $V$ ( $d \times d$ ) are orthogonal and $S$ is diagonal with singular values $s_{1} \geq s_{2} \geq \dots \geq 0$ .

Then $X^{⊤} X = V S^{2} V^{⊤}$ , so the columns of $V$ are the principal directions and the eigenvalues are $λ_{j} = s_{j}^{2} / n$ . The projected data is $Z = X V_{k} = U_{k} S_{k}$ .

Why SVD? It never forms $X^{⊤} X$ , which squares the condition number and can lose precision. It is numerically stable, and truncated / randomised SVD computes only the top $k$ components quickly for huge sparse matrices.

Formula to remember

Variance maximization

v_{1} = ar g ∥ v ∥ = 1 max v^{⊤} Σ v

First component keeps the most variance.

Quick check · +5 XP

Maximizing projected variance and minimizing reconstruction error give…

Your notes

Saved in this browser as you type.