Multiclass classification

With $K>2$ classes there are two main strategies. One-vs-rest (OvR) trains $K$ binary classifiers, each separating one class from all the others, and predicts the class whose classifier is most confident. One-vs-one trains a classifier for every pair and takes a vote.

The direct approach is softmax (multinomial logistic) regression: compute one score $z_k={\mathbf{w}}_k^{\top }\mathbf{x}+b_k$ per class and turn the scores into probabilities:

Softmax outputs are positive and sum to 1; the biggest score gets the biggest probability, and it is shift-invariant (adding a constant to every score changes nothing). It is trained with categorical cross-entropy, $-\log {\hat{p}}_{\text{true class}}$, and it is the final layer of almost every neural-network classifier.