Introduction to Different Activation Functions for Deep Learning

The Idea of Neural Networks was first introduced way back in 1950s, but it wasn't until 2012 that they come to action. Even application of Optimization Algorithm(Gradient Descent) in 2006 by Hinton, wasn't giving good results, it was introduction and usage of Activation functions, which revolutionized Deep Learning Research. There are various kind of Activation Functions that exists, and Some Researchers are still working on finding better functions, which can help networks to converge faster or use less layers etc. Lets go through each of them: The Main Problem we face is because of Saturated Gradients, as the Function ranges between 0 to 1, the values might remain constant, thus the gradients will have very less values. It has all properties of ReLU, plus it will never have dead ReLU problem. We can consider different multiplication factor to form different variations of Leaky ReLU.