Theoretical Limits of Pipeline Parallel Optimization and Application to Distributed Deep Learning

Igor Colin, Ludovic DOS SANTOS, Kevin Scaman

Neural Information Processing Systems 

Distributing the training of deep neural networks can be tackled from several angles.