Activation functions: sigmoid, tanh, ReLU

Sigmoid $\sigma (z)=1/(1+e^{-z})$: range (0, 1), smooth, good for output probabilities. In hidden layers its gradient (at most 0.25) shrinks towards zero for large $|z|$ — saturation — causing vanishing gradients in deep networks. Outputs are not zero-centred.

Tanh $\tanh (z)=\frac{e^z-e^{-z}}{e^z+e^{-z}}$: range (−1, 1), zero-centred, stronger gradients than sigmoid (max 1) — but it still saturates.

ReLU $\max (0,z)$: gradient is 1 for positive inputs and 0 for negative. Cheap, doesn't saturate for positive values, and made deep networks practical. Risk: 'dying ReLU' — a neuron stuck in the negative region never updates. Leaky ReLU ($\max (0.01z,z)$), ELU and GELU fix this.

• Default for hidden layers: ReLU (or GELU in transformers).
• Binary output: sigmoid. Multiclass output: softmax. Regression output: linear (no activation).