The SWAG Algorithm; a Mathematical Approach that Outperforms Traditional Deep Learning. Theory and Implementation

Deep learning allows computational models that are composed of multiple processing layers, to learn very abstract representations of data[LBH15]. There has been reports of many successes using deep neural networks (DNNs) in areas such as computer vision, speech recognition, language processing,drug discovery, genomics, and a host of other areas.[JSA15]. DNNs have allowed us to solve difficult problems and have motivated extensive work to understand their theoretical properties [HDR18]. The process of how to effectively train a DNN is a complicated task and has been proven to be an NP-Complete problem [BR89]. Features such as weight initialization, the nature of 1 activation functions, and network architecture can affect the training process of a neural network [Sch15][HDR18] [RZL18]. In particular, some choices of activation functions or network architectures can cause loss of information or may increase the amount of time needed to train a DNN [HDR18][ZL18][CLP16][LTR17]. The question of how to effectively find the best set of nonlinear activation functions is challenging [CLP16].Some of the well-known nonlinear activation functions are: sigmoid