Objective functions

The objective function is what the optimiser actually minimises: the average loss over the training set, plus any regularisation terms:

Loss is per example; the objective (also called cost) is over the whole dataset. Some objectives are convex — bowl-shaped with a single global minimum, like linear and logistic regression. Neural-network objectives are non-convex, with many local minima and saddle points, yet gradient methods still find good solutions in practice.