Distances vs. divergences

A distance (metric) $d$ must satisfy four rules: non-negativity $d(P,Q)\ge 0$; identity $d(P,Q)=0⟺P=Q$; symmetry $d(P,Q)=d(Q,P)$; and the triangle inequality $d(P,Q)\le d(P,R)+d(R,Q)$.

A divergence only needs the first two: it is non-negative and zero only when the distributions are equal. It may be asymmetric and may break the triangle inequality. KL divergence is the classic example — the 'distance' from P to Q differs from Q to P.

Two big families appear in this topic. F-divergences compare densities point by point through their ratio $p(x)/q(x)$ (KL, JS, χ², TV, Hellinger). Integral probability metrics compare expectations of test functions (Wasserstein, MMD, Kolmogorov). Total variation belongs to both.