KL divergence

The Kullback–Leibler divergence compares a true or reference distribution P with an approximate or predicted distribution Q.

For continuous distributions:

Steps: check both are valid distributions (non-negative, sum to 1); compute $P(i)/Q(i)$; take the log (base 2 or $e$); multiply by $P(i)$; sum over $i$.

Worked example with $P=[0.4,0.6]$, $Q=[0.5,0.5]$: ratios $0.8$ and $1.2$; logs $\approx -0.223$ and $0.182$; weighted $-0.089$ and $0.109$; so $D_{KL}(P\Vert Q)\approx 0.02$.

KL is not symmetric: $D_{KL}(P\Vert Q)\ne D_{KL}(Q\Vert P)$ in general, and it lies in $[0,\infty )$. It shoots to infinity when the distributions do not overlap, which makes it unstable for training GANs.