Forward propagation

Forward propagation computes the network's output by passing the input through the layers in order. For each layer: multiply by the weights, add the bias, apply the activation.

The final layer's output is the prediction, which goes into the loss. During training, the intermediate ${\mathbf{z}}^{(l)}$ and ${\mathbf{h}}^{(l)}$ are stored because backpropagation needs them.

For a mini-batch, stack examples as rows of a matrix and the same formulas become matrix–matrix products, which GPUs compute very fast.