From promise to practice: realizing high-performance decentralized training

With the rapid advancement of deep neural networks (DNNs), distributed training has become the mainstream approach for efficiently scaling up models. One of the most popular algorithms used in data-parallel training is All-Reduce (Li et al., 2020), known for its simplicity and its ability to maintain consistency However, All-Reduce training relies on high-speed network connections and homogeneous computational devices to ensure its efficiency (Zhang et al., 2020; Tandon et al., 2017). Decentralized algorithms, which originally gained attention in the fields of consensus algorithms (Johansson et al., 2007; Shi et al., 2015) and privacy-preserving techniques (Y an et al., 2012), have recently been explored as alternatives to All-Reduce in distributed training, especially in We believe there are several reasons for this: (1) Simply combining the best of each line of work does not necessarily lead to an effective overall system. We propose a simple yet accurate runtime model that quantifies key environmental parameters and estimates potential speedups. We design and analyze a decentralized variant of the Adam optimizer.