Training, validation and test sets
Fit on training data, tune on validation data, report once on test data.
We split the data into three parts with different jobs. The training set (typically 60–80%) is used to fit the parameters. The validation set (10–20%) is used to compare models and tune hyperparameters. The test set (10–20%) is used once, at the very end, to report how well the final model generalises.
Why not just two sets? Every time you pick a model because it scored best on some data, you are slightly fitting to that data. If you tune on the test set, the test score becomes optimistic and no longer an honest estimate. Keeping the test set locked away avoids this information leak.
Splits should be random and, for classification, stratified so each part has the same class proportions. For time series, split by time — train on the past, validate and test on the future — or you will leak information from the future into training.
Formula to remember
Dataset
A supervised dataset: n examples, each a feature vector with its label.