Distributed Learning: A Primer. Behind the algorithms that make Machine…

Distributed learning is one of the most critical components in the ML stack of modern tech companies: by parallelizing over a large number of machines, one can train bigger models on more data faster, unlocking higher-quality production models with more rapid iteration cycles. But don't just take my word for it. Using customized distributed training […] allows us to iterate faster and train models on more and fresher data. Our experiments show that our new large-scale training methods can use a cluster of machines to train even modestly sized deep networks significantly faster than a GPU, and without the GPU's limitation on the maximum size of the model. We sought out to implement a large-scale Neural Network training system that leveraged both the advantages of GPUs and the AWS cloud.