Probabilistic classification

A probabilistic classifier estimates $P(y=k\mid \mathbf{x})$ for every class and predicts the most probable one — the maximum a posteriori (MAP) rule: $\hat{y}=\arg {\max }_{k}P(y=k\mid \mathbf{x})$. If the probabilities are correct, this minimises the error rate (the Bayes optimal classifier).

Generative classifiers model how the data is produced, $P(\mathbf{x}\mid y)$ and $P(y)$, then use Bayes' rule — Naïve Bayes, Gaussian discriminant analysis. Discriminative classifiers model $P(y\mid \mathbf{x})$ directly — logistic regression, neural networks.

Generative models need less data and can handle missing features, but rely on assumptions about $P(\mathbf{x}\mid y)$. Discriminative models usually win with plenty of data because they spend their capacity on the boundary only.