A novel adaptive learning rate scheduler for deep neural networks

Deep learning [8] is becoming more omnipresent for several tasks, including image recognition [22, 29], face recognition [30], and object detection [6]. At the same time, the trend is towards deeper neural networks [12, 9]. Deep convolutional neural networks [15, 16] are a variant that introduce convolutional and pooling layers, and have seen incredible success in image classification [21, 33], even surpassing human-level performance [9]. Very deep convolutional neural networks have even crossed 1000 layers [11]. Despite their popularity, training neural networks is made difficult by several problems. These include vanishing and exploding gradients [7, 3] and overfitting.