Training vs. test error

Training error is the average loss on the data used to fit the model. Test error is the average loss on held-out data the model never saw. The quantity we really care about is the error on the whole distribution, and test error is our estimate of it.

Training error is optimistically biased: the optimiser specifically chose parameters that make it small, so it partly reflects the particular noise in the training sample. A flexible enough model can drive training error to zero by memorising.

The gap between test and training error is the generalisation gap. A small gap with high error means underfitting; a large gap means overfitting.