Deep Learning on GPUs: Successes and Promises

The rise of deep-learning (DL) has been fueled by the improvements in accelerators. Accelerators allow DL models to crunch a large amount of data, which is vital for them to achieve high accuracy. In fact, AlexNet, the famous winner of the ILSVRC 2012 competition, was trained on GPUs. GPU continues to remain the most widely used accelerator for DL applications, due to several of its features, such as high performance, continued improvements in its architecture and software-stack, ease of programming using high-level languages such as CUDA and availability of GPUs in cloud. "Accelerating DL models" is chasing a moving target As DL models are becoming more pervasive and accurate, their compute and memory requirements are growing tremendously.