Experiments with SWISH activation function on MNIST dataset

The important difference is in the negative region of the x-axis. Notice that due to the weird shape of the tail in the negative region, the output from the Swish activation may decrease even when the input is increased. This is very interesting and perhaps unique to Swish. Most activation functions (such as sigmoid, tanh & ReLU) are monotonic i.e. their value never decreases as the input increases (the value may remain same, as is the case for ReLUs when x 0). Intuitively, Swish sits somewhere between linear and ReLU activations. If we consider a simple variation of Swish f(x) 2x*sigmoid(beta*x) where beta is a learnable parameter, it can be seen that if beta 0 the sigmoid part is always 1/2, so f(x) becomes linear.