Regularization

Regularisation adds a penalty for complexity to the training objective, so the optimiser prefers simpler models unless the data strongly supports a complex one:

$\lambda \ge 0$ controls the balance: $\lambda =0$ is plain training; large $\lambda$ forces the weights towards zero. The most common penalties are $\Vert \mathbf{w}{\Vert }_2^2$ (L2, ridge) and $\Vert \mathbf{w}{\Vert }_1$ (L1, lasso).

Regularisation trades a little bias for a big reduction in variance. Other forms include early stopping, dropout, data augmentation and weight decay in neural networks.