Multilayer perceptrons and hidden layers

A multilayer perceptron (MLP) stacks layers of neurons: an input layer (the features), one or more hidden layers, and an output layer. In a fully connected (dense) layer, every neuron receives every output of the previous layer.

Each layer computes ${\mathbf{h}}^{(l)}=\varphi (W^{(l)}{\mathbf{h}}^{(l-1)}+{\mathbf{b}}^{(l)})$. Early layers learn simple features; later layers combine them into more abstract ones.

The universal approximation theorem says a network with one hidden layer and enough neurons can approximate any continuous function on a bounded domain as closely as you like. In practice, deeper networks reach the same accuracy with far fewer neurons, because they reuse features hierarchically.

Design choices: number of layers (depth), neurons per layer (width), activation functions, and the output layer — one linear unit for regression, a sigmoid for binary, a softmax for multiclass.