Wasserstein distance

The Wasserstein-1 distance, or earth mover's distance, imagines each distribution as a pile of earth. It is the minimum total work — mass times distance moved — needed to reshape one pile into the other:

$\Pi (P,Q)$ is the set of all transport plans — joint distributions whose marginals are P and Q. By Kantorovich–Rubinstein duality it equals an IPM over 1-Lipschitz functions: $W_1={\sup }_{\Vert f{\Vert }_L\le 1}{\mathbb{E}}_{P}f-{\mathbb{E}}_{Q}f$.

The key advantage: Wasserstein reflects *how far apart* the distributions are, even when they don't overlap. Two point masses at 0 and $\theta$ have $W_1=|\theta |$ — a smooth, useful gradient — while KL is infinite and JS is stuck at $\log 2$. This is why Wasserstein GAN trains more stably.