Loss functions for networks

The loss turns the network's output into a single number to minimise. It must match the output layer:

• Regression — linear output, mean squared error: $L=\frac{1}{n}{\sum }_i(y_i-{\hat{y}}_i{)}^2$. MAE or Huber if there are outliers.\n- Binary classification — sigmoid output, binary cross-entropy: $L=-[y\log \hat{p}+(1-y)\log (1-\hat{p})]$.\n- Multiclass — softmax output, categorical cross-entropy: $L=-{\sum }_k{y}_k\log {\hat{p}}_k=-\log {\hat{p}}_{\text{true}}$.

Why cross-entropy rather than MSE for classification? Combined with a sigmoid or softmax, its gradient with respect to the scores is simply $\hat{p}-y$ — large when the network is confidently wrong. MSE through a sigmoid has a gradient that vanishes exactly when the network is confidently wrong, so learning stalls.