The reconstruction perspective

Deriving PCA as the projection with the smallest reconstruction error.

An equivalent derivation: find the $k$ -dimensional subspace that reconstructs the data with the smallest squared error. Project each point onto the subspace and map it back: $x^= W_{k} W_{k}^{⊤} x$ .

W_{k} min \frac{1}{n} i \sum x_{i} - W_{k} W_{k}^{⊤} x_{i}^{2} with W_{k}^{⊤} W_{k} = I

By Pythagoras, each point's squared length = squared length of its projection + squared reconstruction error. The total is fixed, so minimising the error equals maximising the projected variance — the same solution: the top- $k$ eigenvectors. The minimum error equals the sum of the discarded eigenvalues, $\sum_{j > k} λ_{j}$ .

Formula to remember

Principal directions

Σ v_{k} = λ_{k} v_{k}

λ_k is the variance along v_k.

Quick check · +5 XP

Before PCA, features on very different scales should be…

Your notes

Saved in this browser as you type.