Lesson 3 of 7 8 min · 130 words
Synchronising updates
Parameter servers, all-reduce, synchronous vs. asynchronous training.
Data-parallel workers must combine their gradients. Two main architectures:
- Parameter server: central servers hold the parameters; workers pull the latest values, compute gradients and push them back. Simple, but the servers can become a bandwidth bottleneck.
- All-reduce: workers exchange gradients directly (e.g. ring all-reduce) so every worker ends up with the average. No central bottleneck — the standard for GPU clusters.
Synchronous training waits for every worker each step, so it behaves exactly like large-batch SGD — but the slowest worker sets the pace. Asynchronous training lets workers update whenever they finish; it is faster per step but uses stale gradients computed from older parameters, which can hurt convergence.
Formula to remember
Federated averaging
Global model = data-weighted average of client models.
Quick check · +5 XP