The closed-form solution

Setting the gradient of $\Vert \mathbf{y}-X\mathbf{w}{\Vert }^2$ to zero gives $X^{\top }X\mathbf{w}=X^{\top }\mathbf{y}$ — the normal equations — and, if $X^{\top }X$ is invertible:

This gives the exact optimum in one step, with no learning rate or iterations. The cost is forming and solving a $d\times d$ system, roughly $O(n{d}^2+d^3)$. That is instant for tens or hundreds of features but impractical for hundreds of thousands.

$X^{\top }X$ is singular when features are perfectly collinear or when $d>n$. Then there are infinitely many solutions; ridge regression's $(X^{\top }X+\lambda I{)}^{-1}$ or the pseudo-inverse fixes it. Numerically, libraries use QR or SVD decompositions rather than computing the inverse directly.