Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training

In this paper, we carefully analyze the AllReduce based setup, propose timing models which include network latency, bandwidth, cluster size and compute time, and demonstrate that a pipelined training with a width oftwocombines thebest ofboth synchronous and asynchronous training.