Dynamic Scheduling of MPI-based Distributed Deep Learning Training Jobs

There is a general trend towards solving problems suited to d eep learning with more complex deep learning architectures trained on larger training sets. This requires longer compute times and greater data parallelizati on or model paralleliza-tion. Both data and model parallelism have been historicall y faster in parameter server architectures, but data parallelism is starting to b e faster in ring architectures due to algorithmic improvements. In this paper, we ana lyze the math behind ring architectures and make an informed adaptation of dynam ic scheduling to ring architectures. To do so, we formulate a non-convex, non-lin ear, NPhard integer programming problem and a new efficient doubling heuristic f or its solution. W e build upon Horovod: an open source ring architecture framew ork over TensorFlow. W e show that Horovod jobs have a low cost to stop and restart an d that stopping and restarting ring architecture jobs leads to faster compl etion times. These two facts make dynamic scheduling of ring architecture jobs fea sible. Lastly, we simulate a scheduler using these runs and show a more than halvin g of average job time on some workload patterns.