Understanding ReLU: The Most Popular Activation Function in 5 Minutes!

Sigmoid and tanh were monotonous, differentiable, and previously more popular activation functions. However, these functions suffer saturation over time, and this leads to problems occurring with vanishing gradients. An alternative and the most popular activation function to overcome this issue is the Rectified Linear Unit (ReLU). The above diagram with the blue line is the representation of the Rectified Linear Unit (ReLU) whereas, the green line is a variant of ReLU called Softplus. The other variants of ReLU include Leaky ReLU, ELU, SiLU, etc., which are used for better performance in some tasks.