Backpropagation

Backpropagation computes the gradient of the loss with respect to every weight in one backward pass, using the chain rule. It starts at the loss and moves backwards layer by layer, reusing results so the cost is about the same as one forward pass.

For layer $l$, define the error signal ${\mathbf{\delta }}^{(l)}=\partial L/\partial {\mathbf{z}}^{(l)}$. Then:

Because gradients are products of many factors, they can vanish (factors < 1, e.g. saturated sigmoids) or explode (factors > 1) in deep networks. ReLU, careful initialisation (Xavier, He), batch normalisation, residual connections and gradient clipping all fight this.