Gradient descent

Gradient descent minimises $J$ by repeatedly taking a step against the gradient:

$\eta$ is the learning rate. Each step moves downhill. For convex objectives with a suitable $\eta$, gradient descent converges to the global minimum; for non-convex ones, it converges to a local minimum or saddle point.

Full-batch gradient descent uses all $n$ examples per step — accurate but slow for big datasets. That motivates stochastic and mini-batch variants.