Expected risk
Average loss over the true (unknown) data distribution.
← Machine Learning learning path0%
04
Foundations
Learning as an Optimization Problem
Training as minimising a loss: risk, gradients and the family of gradient-descent methods.
Not started yet
Start Quiz mastery0%
What you’ll take away
- Learning = choose a model class, a loss, and minimise the empirical risk.
- The gradient points uphill; gradient descent steps against it.
- Mini-batch SGD balances noise and cost and is how nearly all models train.
- The learning rate is the critical hyperparameter — read your loss curves.
Major concepts (syllabus)
Hypothesis/model class; parameters; loss functions; objective functions; empirical risk; expected risk; empirical risk minimization; gradient; partial derivatives; gradient descent; stochastic gradient descent; mini-batch optimization; learning rate; convergence.
Lessons
- Hypothesis and model classThe set of functions a learning algorithm is allowed to choose from.8 min
- ParametersThe numbers that pick one function out of the model class.6 min
- Loss functionsMeasuring the cost of one prediction: squared, absolute, log loss.8 min
- Objective functionsLoss summed over data, plus any regularisation terms.7 min
- Expected vs. empirical riskAverage loss over the true distribution versus over the training sample.9 min
- Empirical risk minimizationChoosing the hypothesis with the lowest training loss.8 min
- Gradients and partial derivativesThe direction of steepest increase of a multivariable function.9 min
- Gradient descentRepeatedly stepping against the gradient.10 min
- Stochastic gradient descentUpdating from one example at a time: cheap but noisy.9 min
- Mini-batch optimizationThe practical middle ground used to train most models.8 min
- The learning rateStep size: too small is slow, too large diverges.8 min
- ConvergenceWhen to stop and how to read loss curves.8 min
Key formulas
Empirical risk
Average loss on the training sample.
ERM
Pick the hypothesis with the lowest training loss.
Gradient descent
Step against the gradient with learning rate η.
Mini-batch gradient
Gradient estimated on a batch B; |B| = 1 is SGD.
Key terms
- Batch size
- Number of examples used to compute one gradient update.
- Early stopping
- Stopping training when validation loss stops improving.
- Empirical risk
- Average loss over the training sample.
- Epoch
- One full pass over the training data.
- Expected risk
- Average loss over the true data distribution — what we really want to minimise.
- Gradient
- Vector of partial derivatives; points in the direction of steepest increase.
- Gradient descent
- Iteratively moving parameters against the gradient to reduce the loss.
- Learning rate
- Step size of gradient-based updates.
- Loss function
- Cost of a single prediction compared with the truth.
- Mini-batch
- Small random subset of the data used for one gradient step.
- SGD
- Stochastic gradient descent: updating from one example or a small batch at a time.
Practice this topic6 questions · earn XP and build your streak