parameter server
Efficient Federated Learning against Heterogeneous and Non-stationary Client Unavailability Ming Xiang
Most prior work either overlooks the potential non-stationarity in the dynamics of client unavailability or requires substantial memory/computation overhead.
Towards Understanding Acceleration Tradeoff between Momentum and Asynchrony in Nonconvex Stochastic Optimization
Tianyi Liu, Shiyang Li, Jianping Shi, Enlu Zhou, Tuo Zhao
Neural Information Processing Systems http://nips.cc/
BML: A High-performance, Low-cost Gradient Synchronization Algorithm for DML Training
Songtao Wang, Dan Li, Yang Cheng, Jinkun Geng, Yanshu Wang, Shuai Wang, Shu-Tao Xia, Jianping Wu
In distributed machine learning (DML), the network performance between machines significantly impacts the speed of iterative training. In this paper we propose BML, a new gradient synchronization algorithm with higher network performance and lower network cost than the current practice. BML runs on BCube network, instead of using the traditional Fat-Tree topology.
Deep Leakage from Gradients
Ligeng Zhu, Zhijian Liu, Song Han
Neural Information Processing Systems http://nips.cc/
Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent
Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, Julien Stainer
We study the resilience to Byzantine failures of distributed implementations of Stochastic Gradient Descent (SGD). So far, distributed machine learning frameworks have largely ignored the possibility of failures, especially arbitrary (i.e., Byzantine) ones.
BML: A High-performance, Low-cost Gradient Synchronization Algorithm for DML Training
Songtao Wang, Dan Li, Yang Cheng, Jinkun Geng, Yanshu Wang, Shuai Wang, Shu-Tao Xia, Jianping Wu
Neural Information Processing Systems http://nips.cc/
Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training
Youjie Li, Mingchao Yu, Songze Li, Salman Avestimehr, Nam Sung Kim, Alexander Schwing
Recently, distributed training with AllReduce operations gained popularity as well.