Ridge regression

Ridge regression is least squares with an L2 penalty: ${\min }_{\mathbf{w}}\Vert \mathbf{y}-X\mathbf{w}{\Vert }^2+\lambda \Vert \mathbf{w}{\Vert }_2^2$. It has a closed-form solution:

Adding $\lambda I$ makes the matrix always invertible, which fixes problems when features are highly correlated (multicollinearity) or when there are more features than examples. Ridge shrinks all weights smoothly towards zero but rarely makes any exactly zero.