Lesson 10 of 12 8 min · 120 words
Mini-batch optimization
The practical middle ground used to train most models.
Mini-batch gradient descent averages the gradient over a small random batch of examples (typically 32–512):
It is the practical middle ground: much less noisy than single-example SGD, much cheaper than full batch, and it maps perfectly onto GPU hardware, which processes a whole batch in parallel almost as fast as one example. Almost all deep learning uses it (and people still call it 'SGD').
Popular improvements build on it: momentum (keep a running average of gradients to roll through small bumps), RMSProp and Adam (adapt the step size per parameter).
Formula to remember
Mini-batch gradient
Gradient estimated on a batch B; |B| = 1 is SGD.
Quick check · +5 XP