Activation Functions Explained - GELU, SELU, ELU and more

During the calculations of the values for activations in each layer, we use an activation function right before deciding what exactly the activation value should be. From the previous activations, weights and biases in each layer, we calculate a value for every activation in the next layer. But before sending that value to the activations of the next layer, we use an activation function to scale the output. Here, we will explore different activation functions. The prerequisite for this post is my last post about feedfordward and backpropagation in neural networks, you would have seen that I briefly talked about activation functions, but never actually expanded on what they do for us. Much of what I talk about here will only be relevant if you have the prior knowledge, or have read my previous post. This article has several ads, which links to Amazon. ML From Scratch earns a small commission, when a purchase is made, using the links provided. It goes back into the site; that is, any fees associated with keeping this website live and producing content.