Why distributed learning

Data and models too large for one machine.

Modern ML often outgrows a single machine. Data can be too large to store or process on one computer (petabytes of logs, billions of images). Models can be too large to fit in one GPU's memory (large language models with hundreds of billions of parameters). And time: training on one GPU might take months.

Distributed learning spreads the work across many GPUs or machines to train faster or train things that would otherwise be impossible.

Sometimes the data is distributed *by nature* and can't be centralised — on phones, in hospitals, in different countries with data-protection laws. Then learning must go to the data, which is federated learning.

Formula to remember

Federated averaging

w_{t + 1} = k = 1 \sum K \frac{n _{k}}{n} w_{t + 1}^{(k)}

Global model = data-weighted average of client models.

Quick check · +5 XP

In federated learning, what leaves the client device?

Your notes

Saved in this browser as you type.