Gradient descent for regression

For large datasets or many features, train with gradient descent instead. The gradient of the mean squared error $J(\mathbf{w})=\frac{1}{n}\Vert X\mathbf{w}-\mathbf{y}{\Vert }^2$ is:

Each step: compute predictions, compute errors $X\mathbf{w}-\mathbf{y}$, multiply by $X^{\top }$, and update $\mathbf{w}\leftarrow \mathbf{w}-\eta \nabla J$. Because the MSE is convex, gradient descent reaches the same solution as the normal equation (with a suitable $\eta$).

Feature scaling is essential here: if one feature ranges in thousands and another in fractions, the loss bowl is a long thin valley and gradient descent zig-zags. Standardising features (zero mean, unit variance) makes it round and convergence fast.