Overfitting
Memorising noise: low training error, high test error.
Overfitting happens when a model learns the noise and quirks of the training data instead of the underlying pattern. Symptoms: very low training error, much higher validation or test error.
It is most likely when the model is very flexible (a high-degree polynomial, a deep tree, a large network), the dataset is small, or training runs for too long. Imagine fitting a degree-15 polynomial through 10 noisy points — it passes through every point but swings wildly between them.
On a training curve, overfitting shows as training loss still falling while validation loss starts rising. The point where validation loss is lowest is where you would want to stop (early stopping).
Formula to remember
Bias–variance decomposition
Expected error = squared bias + variance + irreducible noise.