L2 regularization

L2 regularisation (ridge, weight decay) adds $\lambda \Vert \mathbf{w}{\Vert }_2^2=\lambda {\sum }_j{w}_j^2$ to the loss. Large weights are expensive, so the model spreads influence across features and becomes smoother and less sensitive to small input changes.

For linear regression it has the closed form $\hat{\mathbf{w}}=(X^{\top }X+\lambda I{)}^{-1}{X}^{\top }\mathbf{y}$. The added $\lambda I$ makes the matrix always invertible, so there is always a unique solution — even with correlated features or more features than examples.

In gradient descent, the penalty adds $2\lambda \mathbf{w}$ to the gradient, so each step first shrinks the weights by a constant factor — hence the name weight decay.