Learning from data

Learning from data means we assume there is some unknown function $f$ that maps inputs to outputs, $y=f(x)+\text{noise}$. We never see $f$ directly — only a finite set of examples $(x_1,y_1),\dots ,(x_n,y_n)$. The learning algorithm searches for a model $\hat{f}$ that approximates $f$.

Three ingredients are always present: a model class (the family of functions we allow, e.g. all straight lines), a loss that measures how wrong a prediction is, and an optimisation procedure that picks the member of the model class with the smallest loss on the data.

The real goal is not to do well on the examples we have — memorising them would do that — but to generalise: make good predictions on new inputs drawn from the same source. Everything about validation sets, regularisation and model selection exists to protect generalisation.