We study the resilience to Byzantine failures of distributed implementations of Stochastic Gradient Descent (SGD). So far, distributed machine learning frameworks have largely ignored the possibility of failures, especially arbitrary (i.e., Byzantine) ones.

Neural Information Processing Systems http://nips.cc/

Recently, distributed training with AllReduce operations gained popularity as well.

Most prior work either overlooks the potential non-stationarity in the dynamics of client unavailability or requires substantial memory/computation overhead.

Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

Large-scale distributed training of neural networks is often limited by network bandwidth, wherein the communication time overwhelms the local computation time.

Recently, distributed training with AllReduce operations gained popularity as well.

Neural Information Processing Systems http://nips.cc/

While mini-batching is well understood theoretically [e.g.,