Gradients and partial derivatives

The direction of steepest increase of a multivariable function.

A partial derivative $\partial J / \partial θ_{j}$ measures how $J$ changes when only $θ_{j}$ changes. Stacking all partial derivatives gives the gradient:

\nabla J (θ) = (\frac{\partial J}{\partial θ _{1}}, \dots, \frac{\partial J}{\partial θ _{d}})

The gradient points in the direction of steepest increase of $J$ , and its length tells how steep it is. At a minimum (or any stationary point) the gradient is zero.

For neural networks the gradient is computed efficiently by backpropagation, a systematic application of the chain rule.

Formula to remember

Empirical risk

R^(h) = \frac{1}{n} i = 1 \sum n L (h (x_{i}), y_{i})

Average loss on the training sample.

Quick check · +5 XP

Empirical risk is the average loss over…

Your notes

Saved in this browser as you type.