Foundations

Similarity / Divergence Measures

Measuring how different two probability distributions are — the core of generative modelling.

13 lessons~111 min13 formulas7 questions

Not started yet

Quiz mastery0%

Start

What you’ll take away

Divergences need not be symmetric; metrics must be.
F-divergences (KL, JS, χ², TV, Hellinger) compare density ratios point by point.
IPMs (Wasserstein, MMD, Kolmogorov) compare expectations of test functions.
Wasserstein stays informative when distributions don't overlap — key for stable GANs.

Major concepts (syllabus)

F-divergences: KL, JS, Pearson χ², Neyman χ², Total Variation (TV), Hellinger divergences. Integral Probability Metrics (IPMs): Wasserstein distance, Maximum Mean Discrepancy (MMD), Bounded-Lipschitz (Fortet–Mourier) distance, Cramér distance, Kolmogorov distance.

Builds on 2. Mathematical Foundations

Leads to 16. Advanced Topics

Lessons

Key formulas

F-divergence

Slides

D_{f} (P ∥ Q) = E_{Q} [f (\frac{d P}{d Q})]

Convex f with f(1) = 0; a ratio-based divergence.

KL divergence (discrete)

Slides

D_{K L} (P ∥ Q) = i \sum P (i) lo g \frac{P ( i )}{Q ( i )}

Asymmetric; in [0, ∞); infinite for non-overlapping supports.

KL divergence (continuous)

Slides

D_{K L} (P ∥ Q) = \int P (x) lo g \frac{P ( x )}{Q ( x )} d x

Continuous form of KL.

JS divergence

Slides

D_{J S} (P ∥ Q) M = \frac{1}{2} D_{K L} (P ∥ M) + \frac{1}{2} D_{K L} (Q ∥ M) = \frac{1}{2} (P + Q)

Symmetric; equals log 2 for non-overlapping distributions.

Pearson χ²

χ_{P}^{2} (P ∥ Q) = i \sum \frac{( P ( i ) - Q ( i ) ) ^{2}}{Q ( i )}

Squared differences weighted by Q.

Neyman χ²

χ_{N}^{2} (P ∥ Q) = i \sum \frac{( P ( i ) - Q ( i ) ) ^{2}}{P ( i )}

Squared differences weighted by P.

Total variation

T V (P, Q) = \frac{1}{2} i \sum ∣ P (i) - Q (i) ∣

Largest difference in probability assigned to any event.

Hellinger distance

H^{2} (P, Q) = \frac{1}{2} i \sum (P (i) - Q (i))^{2}

Bounded in [0, 1] and symmetric.

Integral Probability Metric

Slides

E (P, Q) = f \in F sup E_{P} f (x) - E_{Q} f (x)

The function class 𝓕 decides which metric you get.

Wasserstein-1

W_{1} (P, Q) = ∥ f ∥_{L} \leq 1 sup E_{P} f (x) - E_{Q} f (x)

Kantorovich–Rubinstein dual form; used by WGAN.

MMD

MMD^{2} = E [k (x, x^{'})] - 2 E [k (x, y)] + E [k (y, y^{'})]

x, x' ~ P and y, y' ~ Q, with kernel k.

Kolmogorov distance

K (P, Q) = x sup ∣ F_{P} (x) - F_{Q} (x) ∣

Largest gap between CDFs.

Cramér distance

ℓ_{2}^{2} (P, Q) = \int (F_{P} (x) - F_{Q} (x))^{2} d x

Integrated squared gap between CDFs.

Key terms

Divergence: Non-negative measure of difference between distributions, zero only if they are equal; need not be symmetric.
F-divergence: Family of divergences built from a convex f with f(1) = 0 applied to the density ratio.
Hellinger distance: Bounded, symmetric distance between distributions based on square roots of densities.
IPM: Integral probability metric: largest difference in expectations over a class of test functions.
JS divergence: Symmetric, bounded divergence: average KL of each distribution to their mixture.
KL divergence: Σ P log(P/Q): information lost when Q approximates P. Asymmetric.
MMD: Maximum mean discrepancy: distance between kernel mean embeddings of two distributions.
Total variation: Largest difference in probability two distributions assign to the same event.
Wasserstein distance: Earth mover's distance: minimum cost of transporting one distribution onto another.

Full glossary →

Practice this topic7 questions · earn XP and build your streak