Principal directions

The principal directions are the eigenvectors of the covariance matrix: $\Sigma {\mathbf{v}}_j={\lambda }_j{\mathbf{v}}_j$. Ordered by eigenvalue ${\lambda }_1\ge {\lambda }_2\ge \cdots \ge {\lambda }_d\ge 0$, the first direction is the axis of greatest spread, the second is the greatest spread perpendicular to the first, and so on.

Each eigenvalue ${\lambda }_j$ is the variance of the data along ${\mathbf{v}}_j$. The total variance is preserved: ${\sum }_j{\lambda }_j=\mathrm{trace}(\Sigma )=$ the sum of the original feature variances.

The principal components (scores) of an example are its coordinates in this new basis: $z_j={\mathbf{v}}_j^{\top }\mathbf{x}$. The entries of ${\mathbf{v}}_j$ (the loadings) show how much each original feature contributes to that component.