Google open-sources GPipe, a library for efficiently training large deep neural networks

If you're in the business of training large-scale AI systems, good news: Google's got your back. Google's AI research division today open-sourced GPipe, a library for "efficiently" training deep neural networks (layered functions modeled after neurons) under Lingvo, a TensorFlow framework for sequence modeling. It's applicable to any network consisting of multiple sequential layers, Google AI software engineer Yanping Huang said in a blog post, and allows researchers to "easily" scale performance. "Deep neural networks (DNNs) have advanced many machine learning tasks, including speech recognition, visual recognition, and language processing. "[In] GPipe … we demonstrate the use of pipeline parallelism to scale up DNN training to overcome this limitation." As Huang and colleagues explain in an accompanying paper ("GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism"), GPipe implements two nifty AI training techniques.