Covariance matrix
d × d feature covariances.
← Machine Learning learning path0%
14
Unsupervised learning
Dimensionality Reduction
Compressing data into fewer dimensions with PCA while keeping what matters.
Not started yet
Start Quiz mastery0%
What you’ll take away
- High dimensions make data sparse and distances uninformative.
- PCA projects onto eigenvectors of the covariance matrix — directions of most variance.
- Max variance and min reconstruction error give the same solution.
- Use SVD to compute it and explained variance to choose k.
Major concepts (syllabus)
High-dimensional data; curse of dimensionality; feature selection vs. feature extraction; Principal Component Analysis (PCA); covariance matrix; principal directions; variance maximization; reconstruction perspective; SVD; explained variance; visualization
Lessons
- High-dimensional dataImages, text and genomics: thousands of features per example.7 min
- The curse of dimensionalityWhy distances and densities behave strangely in high dimensions.8 min
- Feature selection vs. feature extractionKeeping original features versus building new ones.7 min
- Principal Component AnalysisProjecting onto the directions of greatest variance.10 min
- The covariance matrixHow every pair of features varies together.8 min
- Principal directionsEigenvectors of the covariance matrix.8 min
- Variance maximizationDeriving PCA as the projection that keeps the most variance.9 min
- The reconstruction perspectiveDeriving PCA as the projection with the smallest reconstruction error.9 min
- PCA via SVDThe numerically stable way to compute components.8 min
- Explained variance and visualizationChoosing the number of components and plotting data in 2D.8 min
Key formulas
Principal directions
λ_k is the variance along v_k.
Variance maximization
First component keeps the most variance.
Reconstruction error
Equivalent view: best low-rank reconstruction.
SVD
Columns of V are the principal directions of centred X.
Explained variance ratio
Share of variance captured by component k.
Key terms
- Curse of dimensionality
- Data sparsity and distance concentration that make high-dimensional learning hard.
- Dimensionality reduction
- Representing data with fewer features while keeping most of its information.
- Eigenvector
- A vector whose direction a matrix leaves unchanged: Av = λv.
- PCA
- Principal component analysis: projecting data onto directions of greatest variance.
- SVD
- Singular value decomposition X = USVᵀ; the stable way to compute PCA.
Practice this topic6 questions · earn XP and build your streak