Loss functions

A loss function $L(y,\hat{y})$ measures the cost of a single prediction $\hat{y}$ when the truth is $y$. It turns 'how wrong' into a number the optimiser can minimise.

Squared loss $(y-\hat{y}{)}^2$ — standard for regression; punishes large errors heavily and is smooth. Absolute loss $|y-\hat{y}|$ — more robust to outliers. Huber loss — squared for small errors, absolute for large ones.

0–1 loss — 1 if the class is wrong, 0 otherwise; what accuracy measures, but not differentiable. Log (cross-entropy) loss $-[y\log \hat{p}+(1-y)\log (1-\hat{p})]$ — standard for classification; heavily punishes confident wrong answers. Hinge loss $\max (0,1-ys)$ — used by SVMs.