Data and model parallelism

Data parallelism: every worker holds a full copy of the model and processes a different slice of each mini-batch. Each computes gradients on its slice; the gradients are averaged, $\nabla J=\frac{1}{K}{\sum }_k\nabla J_k$, and every copy applies the same update. It is the most common approach and scales well while the model fits on one device.

Model parallelism: the model itself is split across devices because it is too large for one. Tensor parallelism splits individual layers (parts of each weight matrix on different GPUs); pipeline parallelism puts different layers on different GPUs and streams micro-batches through them like an assembly line.

Large models combine all three ("3-D parallelism"), plus techniques like ZeRO / fully sharded data parallel that split optimiser state and parameters across workers.