Restricted Boltzmann machines

Visible and hidden layers with no within-layer connections, trained with contrastive divergence.

A restricted Boltzmann machine (RBM) keeps the visible and hidden layers but removes all connections within a layer: the graph is bipartite. Its energy is:

E (v, h) = - a^{⊤} v - b^{⊤} h - v^{⊤} W h

The restriction makes hidden units conditionally independent given the visible ones, and vice versa: $P (h_{j} = 1 ∣ v) = σ (b_{j} + v^{⊤} W_{:, j})$ and $P (v_{i} = 1 ∣ h) = σ (a_{i} + W_{i, :} h)$ . A whole layer can be sampled in one parallel step.

RBMs are trained with contrastive divergence (CD-k): start from a data vector, do $k$ (often 1) rounds of alternating Gibbs sampling $v \to h \to v^{'} \to h^{'}$ , and update $Δ W \propto v h^{⊤} - v^{'} h^{'⊤}$ — data statistics minus model statistics.

Formula to remember

Generative model goal

In a GAN, the discriminator tries to…