RelaySum for Decentralized Deep Learning on Heterogeneous Data

Neural Information Processing Systems 

Because the workers only communicate with few neighbors without central coordination, these updates propagate progressively over the network.