Federated averaging
Global model = data-weighted average of client models.
← Machine Learning learning path0%
17
Neural & generative models
Distributed ML
Training across many machines or devices, including when the data cannot leave them.
Not started yet
Start Quiz mastery0%
What you’ll take away
- Data parallelism splits the data; model parallelism splits the network.
- All-reduce and parameter servers combine gradients; sync vs async trades speed for staleness.
- Federated learning keeps data on devices and averages model updates (FedAvg).
- Non-i.i.d. data, unreliable clients and privacy leakage are the main challenges.
Major concepts (syllabus)
Distributed Learning, Federated Learning, Distributed Generative Adversarial Networks (DGANs)
Lessons
- Why distributed learningData and models too large for one machine.7 min
- Data and model parallelismSplitting the data versus splitting the network.9 min
- Synchronising updatesParameter servers, all-reduce, synchronous vs. asynchronous training.8 min
- Federated learningTraining on devices; only model updates leave them.9 min
- Federated averagingAveraging client models weighted by their data size.8 min
- Challenges: privacy and heterogeneityNon-i.i.d. data, unreliable clients and leaking information through updates.8 min
- Distributed GANs (DGANs)Training generators and discriminators across distributed data sources.9 min
Key formulas
Data-parallel gradient
Each worker computes a gradient on its shard; results are averaged.
Key terms
- Federated learning
- Training a shared model on decentralised data; only model updates leave the clients.
Practice this topic5 questions · earn XP and build your streak