Deep Learning: Computational Aspects

Deep learning (DL) is a form of machine learning that uses hierarchical layers of abstraction to model complex structures. DL requires efficient training strategies and these are at the heart of today's successful applications which range from natural language processing to engineering and financial analysis. While deep learning has been available for several decades there were only a few practical applications until the early 2010s when the field has changed for several reasons. The renaissance is due to a number of factors, in particular 1. Hardware and software for accelerated computing (GPUs and specialized linear algebra libraries) 2. Increased size of datasets (Massive Data) 3. Efficient algorithms algorithms, such as stochastic gradient descent (SGD). The goal of our article is to provide the reader with an overview of computational aspects underlying the algorithms and hardware, which allow modern deep learning models to be implemented at scale. Many of the leading Internet companies employ DL at scale Hazelwood et al. [2017]. The most impressive accomplishment of DL is its many successful applications in research and business.