Convergence

Convergence means the parameters (and the loss) have stopped changing meaningfully. Common stopping rules: a fixed number of epochs, the gradient norm falling below a threshold, the improvement in loss falling below a tolerance, or early stopping when validation loss has not improved for a set number of epochs (the *patience*).

Always plot both training and validation loss against epochs. Both falling: keep training. Training falling, validation rising: overfitting — stop and keep the best checkpoint. Both flat and high: underfitting or a learning-rate problem.

For convex problems, theory gives convergence rates — gradient descent reduces the error like $O(1/t)$, or exponentially fast for strongly convex functions. For neural networks there is no such guarantee, so empirical monitoring is essential.