High-dimensional data

Many modern datasets have thousands or millions of features per example: a 256 × 256 colour image has 196,608 pixel values; a document as a bag of words has one feature per vocabulary word; gene-expression data measures 20,000 genes per sample.

High dimensionality brings problems: more parameters to learn, more risk of overfitting, slower training, more memory, and data that's impossible to visualise directly.

Fortunately, real high-dimensional data usually lies near a much lower-dimensional structure (the *manifold hypothesis*). Natural images of faces vary along a few dozen factors — pose, lighting, expression, identity — not 196,608 independent ones.