61162d94822d468ee6e92803340f2040-Supplemental-Conference.pdf

In data-parallel optimization of machine learning models, workers collaborate to improve their estimates of the model: more accurate gradients allow them to use larger learning rates and optimize faster. We consider the setting in which allworkerssample fromthesamedataset, andcommunicate overasparsegraph (decentralized).