Bias–variance decomposition
Expected error = squared bias + variance + irreducible noise.
← Machine Learning learning path0%
03
Foundations
Learning & Generalization
Why doing well on training data is not enough, and how to build models that work on new data.
Not started yet
Start Quiz mastery0%
What you’ll take away
- Training error is optimistic; test error estimates real performance.
- Overfitting = memorising noise; underfitting = too simple to capture the pattern.
- Error = bias² + variance + noise; complexity trades one for the other.
- Regularisation (ridge, lasso) penalises complexity; lasso also selects features.
Major concepts (syllabus)
Dataset; Training vs. test error; generalization; overfitting; underfitting; model complexity; bias–variance trade-off; regularization; Ridge regression; Lasso regression; feature selection
Lessons
- Datasets and samplingTraining data as a sample from an unknown distribution.7 min
- Training vs. test errorWhy training error is an optimistic estimate.8 min
- GeneralizationPerforming well on data the model has never seen.8 min
- OverfittingMemorising noise: low training error, high test error.8 min
- UnderfittingA model too simple to capture the pattern.7 min
- Model complexityHow capacity moves a model between underfitting and overfitting.8 min
- The bias–variance trade-offSplitting error into bias, variance and irreducible noise.11 min
- RegularizationPenalising complexity to improve generalization.8 min
- Ridge regressionLeast squares with an L2 penalty.9 min
- Lasso regressionLeast squares with an L1 penalty that produces sparse weights.9 min
- Feature selectionKeeping only the features that help, by filters, wrappers or Lasso.8 min
Key formulas
Ridge objective
L2 penalty shrinks all weights.
Lasso objective
L1 penalty sets some weights exactly to zero.
Key terms
- Bias (statistical)
- Systematic error from wrong model assumptions; high bias causes underfitting.
- Bias–variance trade-off
- Simpler models have more bias and less variance; complex ones the reverse. Test error is minimised in between.
- Feature selection
- Keeping only a useful subset of the original features.
- Generalization
- Performing well on new data from the same distribution as the training data.
- Lasso
- Least squares with an L1 penalty; drives some weights exactly to zero.
- Overfitting
- Fitting noise in the training data: low training error, high test error.
- Ridge
- Least squares with an L2 penalty; shrinks weights smoothly.
- Underfitting
- A model too simple to capture the pattern: high training and test error.
- Variance (model)
- How much a model changes with different training samples; high variance causes overfitting.
Practice this topic6 questions · earn XP and build your streak