Datasets and sampling

We assume the data comes from an unknown probability distribution $\mathcal{D}$ over pairs $(\mathbf{x},y)$. Our dataset is a sample: $n$ examples drawn independently and identically distributed (i.i.d.) from $\mathcal{D}$.

Everything we learn is an estimate based on this sample. A bigger sample gives a better picture of $\mathcal{D}$; a biased sample gives a distorted one. If you only collect data from one hospital, a model may fail at another with different patients or equipment — this is called distribution shift.

Good sampling practice: make sure the data represents the situations where the model will be used, watch for duplicates that cross the train/test boundary, and keep classes reasonably balanced or account for imbalance.