Computational graphs

A computational graph breaks a calculation into simple operations (nodes) connected by the values flowing between them (edges). Inputs and parameters are leaves; the loss is the final node.

For example, $L=(wx+b-y{)}^2$ becomes: $u=wx$, $v=u+b$, $e=v-y$, $L=e^2$. Each node knows how to compute its output (forward) and the local derivative of its output with respect to its inputs (backward).

This representation is what makes deep learning frameworks work. PyTorch and TensorFlow record the graph as you compute, then traverse it backwards to get every gradient automatically — automatic differentiation.