Must Know Tips for Deep Learning Neural Networks, Part 2

Editor's note: Yesterday we posted the first 4 tips & ticks of this fascinating post. Be sure to have a look before reading the second half. One of the crucial factors in deep networks is activation function, which brings the non-linearity into networks. Here we will introduce the details and characters of some popular activation functions and give advices later in this section. The sigmoid non-linearity has the mathematical form?