Generative modeling

A discriminative model learns $p(y\mid \mathbf{x})$ — enough to classify. A generative model learns the distribution of the data itself, $p(\mathbf{x})$ (or $p(\mathbf{x},y)$), so it can sample new data that looks like the training data.

The goal is a model distribution $p_{\theta }$ close to the data distribution $p_{\text{data}}$, measured by a divergence. Maximum likelihood, for example, is equivalent to minimising $D_{KL}(p_{\text{data}}\Vert p_{\theta })$.

• Explicit-density models write down $p_{\theta }(\mathbf{x})$: autoregressive models (PixelCNN, GPT-style language models), normalising flows, variational autoencoders (VAEs, an approximate density), Boltzmann machines.
• Implicit models only provide a way to sample: GANs transform random noise into data.
• Diffusion models learn to reverse a gradual noising process and power most modern image generators.