Gradient-based learning

Training a network repeats one loop until the validation loss stops improving:

• Take a mini-batch of examples.
• Forward pass: compute predictions and the loss.
• Backward pass: backpropagate to get gradients of the loss with respect to every parameter.
• Update: apply an optimiser step, e.g. $\theta \leftarrow \theta -\eta {\nabla }_{\theta }L$.
• Repeat for all batches (one epoch), then evaluate on the validation set.

Optimisers: plain SGD; SGD with momentum (accumulates a velocity to smooth noisy gradients); Adam (per-parameter adaptive step sizes from running averages of gradients and squared gradients — the most common default).

Supporting tricks: random weight initialisation (He for ReLU), input normalisation, learning-rate schedules, dropout (randomly turning off neurons during training), batch normalisation, early stopping and data augmentation.