Least squares

The residual of example $i$ is $r_i=y_i-{\hat{y}}_i$, the vertical gap between the point and the line. Ordinary least squares (OLS) chooses the weights that minimise the sum of squared residuals:

Squaring makes every residual positive, penalises large errors more than small ones, and gives a smooth convex bowl with a single minimum that can be found exactly. Statistically, least squares is the maximum-likelihood estimate if the noise is Gaussian.

The downside: squaring makes OLS sensitive to outliers — one point far from the rest can drag the line towards it. Robust alternatives use absolute or Huber loss.