Lesson 7 of 10 9 min · 113 words
Variance maximization
Deriving PCA as the projection that keeps the most variance.
PCA can be derived by asking: which unit vector maximises the variance of the projected data?
Using a Lagrange multiplier, . Setting the gradient to zero gives : the optimum must be an eigenvector. The projected variance is then , so we pick the eigenvector with the largest eigenvalue.
Repeating with the constraint that each new direction is orthogonal to the previous ones gives the second, third, … eigenvectors in order.
Formula to remember
Covariance matrix
d × d feature covariances.
Quick check · +5 XP