Why Relu? Tips for using Relu. Comparison between Relu, Leaky Relu, and Relu-6.

A neural network without an activation function is essentially just a linear regression model. The activation function does the non-linear transformation to the input making it capable to learn and perform more complex tasks. Today we will be discussing the most commonly used activation function in the neural networks that is Relu. Relu stands for Rectified Linear Unit. A(x) max(0,x), where x is the output of hidden layer. The ReLu function is as shown above.