Boltzmann machines

Energy-based models with stochastic binary units fully connected to each other.

A Boltzmann machine is a network of stochastic binary units $s_{i} \in {0, 1}$ , symmetrically connected to each other. Each configuration $s$ has an energy:

E (s) = - i < j \sum w_{ij} s_{i} s_{j} - i \sum b_{i} s_{i}

and the probability of a configuration follows the Boltzmann distribution from statistical physics:

P (s) = \frac{e ^{- E (s)}}{Z}, Z = s^{'} \sum e^{- E (s^{'})}

Low-energy configurations are likely. Training adjusts weights so that configurations resembling the data get low energy. Units are split into visible (data) and hidden (latent features).

The problem: the partition function $Z$ sums over $2^{N}$ configurations, and sampling with full connectivity mixes extremely slowly. General Boltzmann machines are therefore almost impossible to train at scale — which led to the restricted version.

Formula to remember

GAN game (general)

Slides

G min D max E (X, G_{θ} (K))

D maximises the discrepancy; G minimises it.

Quick check · +5 XP

A model that can generate new samples resembling the data learns…

Your notes

Saved in this browser as you type.