Overfitting and regularization in regression

Polynomial regression adds powers of the features — $x,x^2,\dots ,x^p$ — and fits a linear model on them. A higher degree $p$ can fit curves, but with high $p$ and little data, the curve wiggles through every point and generalises badly.

The classic demo: 10 noisy points from a sine wave. Degree 1 underfits (a line); degree 3 captures the shape; degree 9 hits every point exactly with zero training error and wild swings between them.

Ridge (L2) shrinks the coefficients and smooths the curve; lasso (L1) can zero out unnecessary polynomial terms. Both are controlled by $\lambda$, chosen by cross-validation.