Activation Functions for Deep Learning

Activation functions play a major role in the learning process of a neural network. So far, we have used only the sigmoid function as the activation function in our networks, but we saw how the sigmoid function has its shortcomings since it can lead to the vanishing gradient problem for the earlier layers. In this blog, we will discuss other activation functions; ones that are more efficient to use and are more applicable to deep learning applications. There are seven types of activation functions that you can use when building a neural network. There is the binary step function, the linear or identity function, there is our old friend the sigmoid or logistic function, there is the hyperbolic tangent, or tanh, function, the rectified linear unit (ReLU) function, the leaky ReLU function, and the softmax function.