The learning rate

The learning rate $\eta$ is the most important hyperparameter of gradient methods. Too small: training is painfully slow and may stall. Too large: steps overshoot the minimum, the loss oscillates or diverges to infinity.

For the simple quadratic $J(w)=\frac{a}{2}{w}^2$, gradient descent converges only if $\eta <2/a$. Steeper problems need smaller steps.

In practice: try values on a log scale (0.1, 0.01, 0.001…), watch the loss curve, and use a schedule — warm up, then decay (step, exponential or cosine). Adaptive optimisers like Adam are less sensitive but still need a sensible $\eta$ (0.001 is a common default).