Activation Functions and their purpose: Binary, Linear, ReLU, Sigmoid, Tanh and Softmax

In the context of a neural network an activation function defines the output of a node/neuron, they could be classified into these categories: Ridge activation functions, Radial activation functions and Folding activation functions. For this article we will be looking at Ridge activation functions. Binary step function is a threshold based activation function meaning that if the input crosses a certain value the neuron is activated and if it goes below that value the neuron is deactivated, this function can be used in tasks of binary classification, This activation function is not suitable at all in the case of non-linearity (most of problem domains). Also, since the network is not differentiable, gradient-based training is not possible. As you can see here our function is directly proportional to the weighted sum of neurons: f(x) x.