Secret Sauce behind the beauty of Deep Learning: Beginners guide to Activation Functions

Activation functions are functions which take an input signal and convert it to an output signal. Activation functions introduce non-linearity to the networks that is why we call them non-linearities. Neural networks are universal function approximators and deep Neural Networks are trained using backpropapagation which requires differentiable activation functions. Backpropapagation uses gradient descent on this function to update the network weights. Understanding activation functions is very important as they play a crucial role in the quality of deep neural networks.