The bias–variance trade-off

For squared loss, the expected test error at a point splits into three parts:

Bias is the error from wrong assumptions — how far the average model (over many possible training sets) is from the truth. Variance is how much the model changes when trained on a different sample. Noise ${\sigma }^2$ is irreducible: no model can predict it.

Simple models have high bias and low variance; complex models have low bias and high variance. Reducing one tends to increase the other — the trade-off. The goal is the complexity that minimises their sum.